toga-ai 1.0.1109 → 1.0.1111

package/knowledge/1.0/apps/api/INDEX.md

@@ -3,3 +3,4 @@
 | Doc | Summary |
 |-----|---------|
 | [Talos Ticket API (1.0 api POST/PUT /ticket) — Desk tickets + file uploads from Talos Voice](features/talos-ticket-endpoint.md) | 1.0 `api` route that Talos Voice calls to create (`POST /ticket`) and update (`PUT /ticket/{ticketNumber}`) TOGa Desk tickets and attach call files; open before |
+| [Deploying API 1.0 to Elastic Beanstalk (PHP 8.5 / Amazon Linux 2023)](workflows/deploy-to-elastic-beanstalk-al2023.md) | How the 1.0 `api` deploys on EB PHP 8.5 / AL2023 (Apache proxy) — new prod env + pipeline, s3fs 1.97 mount over IMDSv2, cross-account `asifiles`, php-fpm ini ru |

package/knowledge/1.0/apps/api/features/talos-ticket-endpoint.md

@@ -6,14 +6,15 @@ project: API 1.0
 client: shared
 type: feature
 status: active
-updated: 2026-10-20
+updated: 2026-10-22
 owners: ["jcardinal"]
 files:
   - api/_/app/route/ticket.php
   - api/_/app/error.php
-  - api/ebs/create-s3fs-credentials-file.php
-  - api/ebs/mount-s3fs-folders.php
+  - api/api.php
+  - api/.platform/hooks/predeploy/10-mount-s3fs.sh
 related:
+  - ../workflows/deploy-to-elastic-beanstalk-al2023.md
   - ../../togadesk/features/s3fs-asifiles-file-storage.md
   - ../../library/features/app-cloud-aws-credentials.md
   - ../../togadesk/features/email-to-ticket-intake.md
@@ -33,14 +34,16 @@ Server side of the Talos ticket contract (client side: [ticket API integration](
 - `assertStorageIsMounted()` runs **before any ticket/reply/file row is written**: compares `stat()` dev of `internal.storage_folder` vs its parent; same dev = mount down → `RuntimeException`. Skipped in dev mode.
 - Per file: `fileAlreadySaved($replyId, $filename)` skips a name already on that reply (raw or `<id>-name`) — Talos re-sends. Then save `files` row → write `<storage_folder>/togadesk-<env>/uploads/<id>-<name>` (= `s3://asifiles/togadesk-<env>/uploads/<id>-<name>` via the mount) → rename row to `<id>-<name>`. Write failure (local error handler turns the warning into an exception) deletes the row.
 - `_/app/error.php` wraps its own `App_Cloud` S3 copy in `try/catch (\Throwable)` so the `Logs.Errors` row is always written.
-- s3fs auth = `-o iam_role=auto` (instance role); `ebs/create-s3fs-credentials-file.php` only deletes `/etc/passwd-s3fs`. See [asifiles s3fs mount](../../togadesk/features/s3fs-asifiles-file-storage.md).
+- s3fs auth = `-o iam_role=auto` (instance role), mounted by `.platform/hooks/predeploy/10-mount-s3fs.sh` (s3fs 1.97, cross-account `asifiles`) — see [api deploy](../workflows/deploy-to-elastic-beanstalk-al2023.md) and [asifiles s3fs mount](../../togadesk/features/s3fs-asifiles-file-storage.md).
 
 ## Gotchas
 
 - **`App_Cloud` in api throws `Error`, not `Exception`** (no AWS SDK, see [App_Cloud](../../library/features/app-cloud-aws-credentials.md)). `catch (Exception)` misses it. TRUE-82935: `copyFileToS3` after the mount write threw → row never renamed, request failed, and `error.php` itself crashed so nothing reached `Logs.Errors`.
 - **A failed PUT repeats hourly.** Talos re-sends at ~:00:49 and resets the reply timestamp each time → one more raw-named `session_report_*.txt` row per hour per ticket. Cleanup `dbchanges/TOGaDeskSupport/JC/2026-10-19.sql` ran on prod 2026-10-20 after the fix deployed: deleted 4,481 rows (header said 4,219 — rows kept growing until deploy); 0 bad rows remain. A cleanup like this must run **after** the fix deploys, else rows come back.
-- **Security, open (values not recorded):** `config.*.ini` sits in the web root with SES SMTP credentials and no deny rule; the old api s3fs IAM key is still in git history (key itself deleted 2026-10-12, TRUE-82705).
+- **PHP 8.x: Talos JSON fields can be null.** `App_Error` exits on any warning/deprecation, so cast to `(string)` before `trim` / `base64_decode` / `stripos` (done in `ticket.php` and `api.php` document/securityToken).
+- **Security, open (values not recorded):** `config.*.ini` in the web root is now blocked by an `.htaccess` deny rule (TRUE-83054), but the secrets in it still need rotating; the old api s3fs IAM key is still in git history (key itself deleted 2026-10-12, TRUE-82705).
 
 ## Change history
+- 2026-10-22 — PHP 8.5 / AL2023 move: null-safe `(string)` casts on Talos fields, `curl_close` removed, config deny rule (TRUE-83054) (jcardinal)
 - 2026-10-20 — api PR #39 deployed; first correctly named Talos file row id 615850, no raw-named rows or duplicates after it; cleanup run (TRUE-82935) (jcardinal)
 - 2026-10-19 — Created: removed `copyFileToS3` from uploads (mount write is the S3 write), mount check before any row, per-reply duplicate guard, `error.php` Throwable guard, s3fs `iam_role=auto` (TRUE-82935, branch, not deployed) (jcardinal)

package/knowledge/1.0/apps/api/workflows/deploy-to-elastic-beanstalk-al2023.md

@@ -0,0 +1,67 @@
+---
+title: Deploying API 1.0 to Elastic Beanstalk (PHP 8.5 / Amazon Linux 2023)
+framework: "1.0"
+repo: api
+project: API 1.0
+client: shared
+type: workflow
+status: active
+updated: 2026-10-22
+owners: ["jcardinal"]
+files:
+  - api/.ebextensions/006_mount-s3fs.config
+  - api/.ebextensions/009_setup_phpini.config
+  - api/.ebextensions/020_setup_git_libraries.config
+  - api/.ebextensions/030_dbchanges.config
+  - api/.ebextensions/050_register_instance_to_shared_application_load_balancer.config
+  - api/.platform/hooks/predeploy/10-mount-s3fs.sh
+  - api/.platform/hooks/postdeploy/01-restart-php-fpm.sh
+  - api/.platform/httpd/conf.d/timeouts.conf
+  - api/.htaccess
+  - api/ebs/s3fs.prod.json
+related:
+  - ../features/talos-ticket-endpoint.md
+  - ../../tools/workflows/deploy-to-elastic-beanstalk-al2023.md
+  - ../../walmart-activate/workflows/deploy-to-elastic-beanstalk-al2023.md
+  - ../../togadesk/features/s3fs-asifiles-file-storage.md
+  - ../../library/features/php-8-5-runtime-compatibility.md
+---
+
+How the 1.0 `api` deploys on EB PHP 8.5 / AL2023 (Apache proxy) — new prod env + pipeline, s3fs 1.97 mount over IMDSv2, cross-account `asifiles`, php-fpm ini rules, cutover/rollback; open before touching `.ebextensions`/`.platform` or deploying api.
+
+## Summary
+
+TRUE-83054 moved `api` off EB "PHP 7.2 / 64bit Amazon Linux 2.8.9" (AL1, Apache + mod_php) to "PHP 8.5 / Amazon Linux 2023 4.13.13" with `ProxyServer: apache`. Shared AL2023 rules (ini via cfn-init `files:`, fpm overrides `/etc/php.d`, clone fails the deploy, `/var/app/staging`) live in the [Tools deploy doc](../../tools/workflows/deploy-to-elastic-beanstalk-al2023.md) — not repeated here. This doc holds only what differs for api.
+
+- Env `togatech-legacy-api-production`, EB app `API`, us-east-1, production account 654654170868. Old AL1 env kept as rollback; cutover = CNAME swap.
+- Pipeline `Legacy-API-Production` (us-east-1), copied from `Legacy-Store-Production`: role `service-role/AWS-CodePipeline-Service`, artifacts `true-elasticbeanstalk-deployments-1`, CodeStar source `agilantsolutions/api` branch `_production`, Manual approval stage, EB deploy.
+
+## Steps
+
+1. `009_setup_phpini.config` — sets `ProxyServer: apache`; writes `/etc/php.d/application.ini` (cfn-init `files:`); `memory_limit` / `max_execution_time` go in `aws:elasticbeanstalk:container:php:phpini` because the fpm pool overrides `/etc/php.d` for them.
+2. `005` CVE-2026 stopgap kept; `post_max_size` / `upload_max_filesize` = 50M (8M broke Talos base64 uploads). Deploy guard path is `/var/app/staging`. Installs + enables `cronie`.
+3. `006_mount-s3fs.config` builds s3fs **1.97** from the pinned GitHub tarball, sha256 checked (mismatch fails the deploy). Needs `fuse3` / `fuse3-devel`; `libcurl-devel` via `dnf install --allowerasing`.
+4. `.platform/hooks/predeploy/10-mount-s3fs.sh` (identical copy in `.platform/confighooks/predeploy/`) mounts per `ebs/s3fs.<ENVIRONMENT>.json` (env from `get-config`), webapp uid/gid, `iam_role=auto`, cache `/var/cache/s3fs`; then a write/read/delete test as webapp, else `exit 1`. Also creates the cache dir (mkdir/chown).
+5. `020_setup_git_libraries.config` (bash, `exit 1` on failure) clones `library` and `dbchanges` `_production`. Token = EB env property `GITHUB_TOKEN` read with `get-config`, passed via `GIT_ASKPASS` — never in the URL or log.
+6. `030_dbchanges.config` runs the dbchanges runner `leader_only`.
+7. `050` registers the instance to the shared ALB (IMDS host `169.254.169.254`); skips env names containing `-prod`.
+8. `.platform/hooks/postdeploy/01-restart-php-fpm.sh` restarts php-fpm and checks the live ini values.
+9. Apache conf in `.platform/httpd/conf.d/`: `enforce_https.conf` (from api2), `timeouts.conf` (`Timeout` / `ProxyTimeout` 900 — ALB `IdleTimeout` must also be 900), `hostlog.conf`.
+10. Root `.htaccess`: `FilesMatch` deny for `config.*`, `.ini/.env/.json/.sh`, dotfiles, `test.php`; `manual/.htaccess` = `Require all denied`.
+
+**Cross-account `asifiles`** (bucket in us-west-2, legacy account 502614707982): bucket policy allows role `arn:aws:iam::654654170868:role/ElasticBeanstalk-EC2-Instance-Profile` ListBucket / GetBucketLocation + Get/Put/Delete/PutObjectAcl; that shared role has inline policy `asifiles-readwrite`. Bucket has no ownership controls, so `public-read` ACLs work.
+
+## Gotchas
+
+- **`php_value` in `.htaccess` = "Invalid command" 500 under php-fpm.** Move each value to `application.ini` or the `phpini` option settings.
+- **AL1 hooks are dead on AL2023.** Anything under `/opt/elasticbeanstalk/hooks/appdeploy/...` (s3fs mount, cache folder, cron, tests) never runs. Use `.platform/hooks/*` (+ `confighooks` copy for config-only deploys).
+- **s3fs 1.84 cannot read the instance role over IMDSv2** (AL2023 EB sets `DisableIMDSv1=true`) → mount fails. Use 1.97 (1.96+ needs fuse3). `nonempty` is ignored on fuse3. Keep the cache off `/tmp` (tmpfs on AL2023).
+- **`libcurl-devel` conflicts with preinstalled `libcurl-minimal`.** Use `dnf install --allowerasing`; `dnf swap` fails on rerun.
+- **Same `container_commands` key in two configs = one silently wins.** `035`'s graceful-restart cron was already dead this way (overwritten by `050`'s duplicate key).
+- **`iam simulate-principal-policy` with several resources reports for the first one only.** Simulate object actions against an object ARN.
+- **Disable the old account's pipeline before merging api into `_production`.** It still deploys `_production` to the AL1 rollback env and would push AL2023 config there, breaking the rollback.
+- Prefork tuning (old `001`) does nothing under event MPM + php-fpm. Removed: `010` redis (api has `useSessions=false`; `$_SESSION` is per-request only), `035` cron, `040` tests, `codecept.phar`, `tests/`, `ebs/git.json`, old `setup_git_libraries.php` path (needed a `CodePipelineEnvironments` row).
+- **Security, open (values not recorded):** `config.prod.ini` in the repo holds DB / SMTP / cert passwords (now blocked from the web by the deny rule; still need rotating); an old GitHub token is in git history (`ebs/git.json`, also tools `020`); `test.php` has a hardcoded client token. Developer will rotate.
+
+## Change history
+- 2026-10-22 — Created: AL1 → PHP 8.5 / AL2023 with Apache proxy, new prod account env + pipeline, s3fs 1.97 over IMDSv2, cross-account `asifiles` (TRUE-83054) (jcardinal)

package/knowledge/1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md

@@ -6,7 +6,7 @@ project: Database Changes
 client: shared
 type: workflow
 status: active
-updated: 2026-10-19
+updated: 2026-10-22
 owners: [sking, jcardinal]
 files:
   - dbchanges/index.php
@@ -36,7 +36,7 @@ Two rules dominate everything below:
 1. Confirm the target schema/environment (`Core/` = legacy V1 `Core`).
 2. Branch per the git-workflow rules (`fix/...`, `feature/...`) — never commit to `_production`.
 3. Write **one file, commentary first, statements last**, no trailing comment chunk, no explicit `COMMIT`, and no `DELIMITER`/stored-routine constructs.
-4. Wrap any risky data change in the derived-table preflight guard (below) so a drifted before-state is a no-op rather than a wrong write.
+4. Wrap any risky data change in the derived-table preflight guard (below) so a drifted before-state is a no-op rather than a wrong write. One-time repairs also follow *Rules for a one-time data repair*.
 5. Open a PR into `_production` (or the target `_<env>` branch).
 6. After merge: `_<env>` branches apply within ~2 min; **`_production` waits for the next worker deploy** — verify with a read-only query afterward rather than assuming. **Merging the PR applies nothing by itself** — `dbchanges` has no CI, no workflow, no runner of its own; every application is driven from the worker tier.
 7. If the change spans more than one top-level (database) folder, verify **each database separately** — see the silent-skip trap below.
@@ -84,6 +84,14 @@ WHERE t.id IN (<ids>)
       ) AS preflight) = <expected-row-count>;
 ```
 
+### Rules for a one-time data repair (TRUE-83064, dbchanges#270)
+
+1. **A multi-direction move is ONE `UPDATE`** — `SET col = CASE id WHEN … THEN … END` over the whole id set, under one preflight that requires the **exact before-state count of all rows**. Two UPDATEs can half-apply; one statement is all-or-nothing.
+2. **Guard a dependent follow-up UPDATE on the previous step's after-state** (its preflight counts rows in the state step 1 should have left). If step 1 no-op'd, step 2 no-ops too.
+3. **Limit the forward UPDATE to the exact id list** (never a broad `WHERE` like "all units on item X"). The rollback file uses the **same ids** and pins the **current** (after) values.
+4. **Never raise an SQL error on guard mismatch** (no forced-error tricks). The runner `exit()`s on any error and blocks every file queued after it. A mismatch must leave the file a quiet no-op.
+5. **A no-op still commits the `_dbchanges` row**, so the file never re-runs. Re-read the data after apply; if it no-op'd, ship a new file.
+
 ### One top-level folder = one DATABASE (and the silent-skip trap)
 
 The repo's top level is a list of **database names**, not modules: `Core`, `TOGaDeskSupport`, `Common`, `Logs`, `Bridge_NetSuite`, … The runner walks every top-level directory and treats the **folder name as the database to connect to** (`index.php` ~L55–60), then — for each folder — scans `config.<env>.ini` and uses the **first `[database*]` group it finds** (~L76–80; in `config.prod.ini` that is `[database_main]`) for hostname/username/password. So *all* folders are attempted with **one** set of credentials against one host, with only the DB name varying.
@@ -116,6 +124,7 @@ The repo ships `config.alpha/beta/demo/hotfix/prod/stage/test/worker.ini`, there
 - **⚠ SECURITY — hardcoded GitHub token.** `worker/crons/infrastructure/execute_dbchanges.php` contains a **plaintext GitHub personal access token** in the small config block near the top of the file (~L14–18), and interpolates it into a `shell_exec()` git-clone command line — which also exposes it in the worker host's process listing. Treat the token as **compromised**: rotate it, move it to config/env (`config.<env>.ini`), and pass it without embedding it in an argv string. No credential value is recorded here by design.
 
 ## Change history
+- 2026-10-22 — TRUE-83064 (dbchanges#270, CodeRabbit review): one-time data-repair rules — one CASE UPDATE with exact-count preflight, chained after-state guards, exact-id forward + pinned rollback, no SQL error on mismatch, no-op still commits `_dbchanges`. (sking)
 - 2026-10-19 — Comment lines ending in `;` also split the file; verify SELECTs in the header without `;` (TRUE-82935) (jcardinal)
 
 ## Related

package/knowledge/1.0/apps/library/features/php-8-5-runtime-compatibility.md

@@ -7,9 +7,11 @@ client: shared
 type: feature
 status: active
 updated: 2026-10-22
-owners: ["rgirish"]
+owners: ["rgirish", "jcardinal"]
 files:
   - library/app/error.php
+  - library/app/date.php
+  - library/app/database.php
   - library/app/error/capture.php
   - library/app/apitransaction.php
   - library/app/string.php
@@ -21,6 +23,7 @@ related:
   - error-capture-1-0.md
   - ../../walmart-activate/workflows/deploy-to-elastic-beanstalk-al2023.md
   - ../../tools/workflows/deploy-to-elastic-beanstalk-al2023.md
+  - ../../api/workflows/deploy-to-elastic-beanstalk-al2023.md
 ---
 
 Which library calls/signatures fatal a 1.0 app on PHP 8.4/8.5, the guard pattern that stays PHP 7.2 safe, what is fixed and what is still open; open before moving any 1.0 app to PHP 8.4+.
@@ -42,6 +45,13 @@ Fix patterns (TRUE-83059, commit `d28f9b23`):
 
 Implicit-nullable fixes so far: `App_Cloud::getDatabaseInstances`, `App_Cloud::sendOverAs2Connector`, `App_Email::setQueueForSending`, `App_Model_ServiceRequest::triggerStatusEvent`.
 
+Null guards (TRUE-83054, commit `04a2293c`, landed directly on `_production`) — same results as on 7.2:
+- `App_Date::convertToEpoch(null|'')` → `false`, then normal flow; `isValid(null)` → `false`; `convertMDYYYYtoSQLDate(null)` → `null`.
+- `App_Database::fetchOne` → `null` on an out-of-range row or missing key.
+- `App_Database::sqlProtect(null)` → `''`.
+
+**Main 8.x break classes in 1.0 app code** (found moving `api`): `null` into internal string functions (`trim`, `base64_decode`, `stripos` — deprecated 8.1) → cast `(string)`; reading keys of an empty `fetchRow` result (warning since 8.0) → default the array; NULL DB dates into `App_Date` → guard first. Fix in library when the root cause is in a library helper.
+
 **Find the rest:** run `php -l` with a PHP 8.4+ binary and `-d display_errors=1` on each file — it prints compile-time deprecations per file. Runtime-only ones (`curl_close`, `setAccessible`, `utf8_decode`) need a grep.
 
 ## Gotchas
@@ -53,6 +63,8 @@ Implicit-nullable fixes so far: `App_Cloud::getDatabaseInstances`, `App_Cloud::s
   - `App_Model_ServiceRequest::ajaxLoadItemDialog` uses `utf8_decode` (deprecated 8.2).
 - An app only dies on files it loads — Walmart Activate runs on 8.5 with the open list above because it never loads those classes. A new 8.5 app must re-check its own load path.
 - Local dev on PHP 8.2 does not show 8.4/8.5 deprecations — test with an 8.5 binary.
+- **PHP 8 string-to-number comparison can turn a loop infinite.** `api` subscription: a non-numeric quantity compared to an int looped forever on 8.x. Cast `(int)` before the loop.
 
 ## Change history
+- 2026-10-22 — `App_Date` / `App_Database` null guards; 8.x break classes from the `api` move (TRUE-83054) (jcardinal)
 - 2026-10-22 — Created: guard pattern, fixed list, open list from the Walmart Activate 8.5 move (TRUE-83059) (rgirish)

package/knowledge/1.0/apps/togadesk/features/s3fs-asifiles-file-storage.md

@@ -6,7 +6,7 @@ project: TOGa Desk
 client: shared
 type: feature
 status: active
-updated: 2026-10-20
+updated: 2026-10-22
 owners: ["mhammontree", "snaredla", "jcardinal"]
 files:
   - togadesk/ebs/mount-s3fs-folders.php
@@ -38,13 +38,14 @@ Every Desk file read from `/var/app/asifiles/...` (Knowledge Base docs `knowledg
 
 - **Triage: `file_get_contents(/var/app/asifiles/...): No such file or directory` on a 1.0 EB app while the object IS in S3 = broken s3fs mount** (credentials or dropped mount), not missing data or orphan DB rows. Check in order: object in S3; on the instance `ls /var/app/asifiles/<path>`; mount uses `-o iam_role=auto`; instance role has `asifiles` access.
 - TRUE-82718 cause: `create-s3fs-credentials-file.php` wrote a hardcoded long-lived IAM key into `/etc/passwd-s3fs`; that key was deleted in the TRUE-82705 cleanup (2026-10-12) → Desk KB red box. Never put a key back in the mount; use the instance role.
-- The mount hook uses the **Amazon Linux 1** hooks path (`/opt/elasticbeanstalk/hooks/...`) — it will not run on AL2023. Rewrite as a platform hook when migrating.
+- The mount hook uses the **Amazon Linux 1** hooks path (`/opt/elasticbeanstalk/hooks/...`) — it will not run on AL2023. Rewrite as a platform hook when migrating. Also **s3fs 1.84 cannot read the instance role over IMDSv2** (AL2023 default) — move to 1.97 + fuse3. Working pattern: [api deploy doc](../../api/workflows/deploy-to-elastic-beanstalk-al2023.md) (TRUE-83054).
 - **Open, not fixed — `template/pages/documents/viewdocument.php`:** `$_GET['id']` goes raw into the file path (path traversal, e.g. `../../../current/config.prod.ini`) and into unescaped SQL on `kb_versioning` and `kb_documents` (views UPDATE). A missing file red-boxes (`App_Error` exits on the warning) instead of a clean message. Fix: cast id to `(int)`; guard the read and show "document not available". Needs its own ticket.
 - The KB path is hardcoded to `knowledgebase-prod` (env-based line commented out), so non-prod reads prod KB files.
 - **Writing into the mount IS the S3 upload.** A file written under `/var/app/asifiles/<prefix>/` lands at `s3://asifiles/<prefix>/`; do not also `App_Cloud::copyFileToS3` it. Used by the email cron (worker `ontrack/desk/uploads` → `/var/app/asifiles/togadesk-prod/uploads`, `worker/ebs/setup_symbolic_links.php` + `links.worker.json`) and the 1.0 `api` Talos uploads ([Talos ticket API](../../api/features/talos-ticket-endpoint.md), same `iam_role=auto` setup since TRUE-82935). A write while the mount is down goes to local disk hidden under the mount point — check the mount first (`is_link`, or compare `stat()` dev with the parent).
 - **Open, not fixed — `odp-togadesk-prod` (Office Depot) deploys fail.** TOGaDesk-Production deploys of PR #624 and #625 (2026-10-16) failed there: `command 04-make_s3fs in .ebextensions/006_mount-s3fs.config failed` (s3fs 1.84 source build: virtual memory exhausted). `agilant-togadesk-prod` and `imhouston-togadesk-prod` succeeded. ODP instances may run mixed versions. 2026-10-20: `odp-togadesk-prod` instance type and memory limits raised (developer, ops) — **not yet verified** by a successful TOGaDesk-Production release; check the next release's `04-make_s3fs` step on ODP.
 
 ## Change history
+- 2026-10-22 — AL2023 note: s3fs 1.84 fails over IMDSv2; api now runs 1.97 from a `.platform` hook (TRUE-83054) (jcardinal)
 - 2026-10-20 — ODP instance size + memory raised for the s3fs build OOM; awaiting a successful release (TRUE-82935) (jcardinal)
 - 2026-10-19 — Mount write = S3 upload (email cron, api); ODP s3fs build failure on deploy (TRUE-82935) (jcardinal)
 - 2026-10-13 — Created: s3fs mount moved from the dead IAM key to `-o iam_role=auto`; check_s3fs/register scripts assume `WorkloadsRuntime` (TRUE-82718, togadesk PR #622) (snaredla, mhammontree)

package/knowledge/1.0/apps/tools/workflows/deploy-to-elastic-beanstalk-al2023.md

@@ -29,7 +29,7 @@ How the Tools 1.0 app deploys/boots on Elastic Beanstalk (PHP 8.5 / Amazon Linux
 
 ## Summary
 
-The Tools 1.0 app runs on Elastic Beanstalk `PHP 8.5 on 64bit Amazon Linux 2023/4.13.1 (aarch64)`. The EB platform was upgraded from Amazon Linux 2 (AL2) to Amazon Linux 2023 (AL2023), which changes package names, the proxy (nginx instead of Apache), the PHP ini scan directory, and how credentials reach the instance. The `.ebextensions/*.config` files run alphabetically and are merged by EB; getting them wrong leaves the app booting without `/var/www/library` and serving 500s on every request. This is the migration checklist and the durable AL2023 gotchas learned doing it.
+The Tools 1.0 app runs on Elastic Beanstalk `PHP 8.5 on 64bit Amazon Linux 2023/4.13.1 (aarch64)`. The EB platform was upgraded from Amazon Linux 2 (AL2) to Amazon Linux 2023 (AL2023), which changes package names, the default proxy (nginx — unless the env sets `ProxyServer: apache`, see the correction under *ignoreErrors*), the PHP ini scan directory, and how credentials reach the instance. The `.ebextensions/*.config` files run alphabetically and are merged by EB; getting them wrong leaves the app booting without `/var/www/library` and serving 500s on every request. This is the migration checklist and the durable AL2023 gotchas learned doing it.
 
 ## Steps
 
@@ -81,6 +81,8 @@ EB merges every `.config` alphabetically; two configs with **identical** `contai
 ### ignoreErrors on non-fatal infrastructure commands
 Several `container_commands` write to system paths that do not exist on AL2023 (e.g. `/etc/httpd/conf.d/` — AL2023 EB uses **nginx**, not Apache, as the reverse proxy). These were marked `ignoreErrors: true` in `004`, `006`, `007`, `009`, and `050` so a non-fatal failure does not block the deploy. This also covers `006`'s `01_credentials` step: modern EB uses IAM roles, so `AWS_SECRET_KEY` / `AWS_ACCESS_KEY_ID` are not injected into `$_SERVER` and the s3fs credential write fails cleanly. Use `ignoreErrors` only on genuinely non-fatal infra setup — the git clone (`020`) deliberately does **not** use it (see gotchas).
 
+> **Correction (TRUE-83054):** "AL2023 uses nginx" is only true for the default nginx proxy. With `ProxyServer: apache`, `/etc/httpd` exists, but custom Apache config belongs in `.platform/httpd/conf.d/*.conf`, not written by `container_commands`. Check which proxy this env runs before trusting the `ignoreErrors` above. See the [api deploy doc](../../api/workflows/deploy-to-elastic-beanstalk-al2023.md).
+
 ### Prod is multi-instance — never store cross-request state on local disk
 Prod `tools` runs **multiple EB instances behind a load balancer** (see `050_register_instance_to_shared_application_load_balancer.config`). Any state written on one request may be read on another instance on the next request. Consequences:
 - **`/var/www/cache` is per-instance LOCAL disk, not shared.** It is created and `chmod 777`ed per instance by `ebs/setup_export_cache_folders.php`; it is **not** an s3fs/shared mount, so it cannot hold state that a later request must read. Symptom seen: a Preview→Send flow that cached the parsed job to `/var/www/cache` failed with "That job has expired" because Preview hit instance A and the Send AJAX was load-balanced to instance B.
@@ -116,9 +118,13 @@ The deploy clones `agilantsolutions/library` and `resources` via `020_setup_git_
 - **Native PECL extensions: use `php8.5-pecl-redis6`, exclude the php8.1 serializer deps, and copy the `.so` into the platform `extension_dir`.** A plain `dnf install` pulls `php8.1-common` which conflicts with `php-common-8.5.6`; install with `--setopt=install_weak_deps=False --exclude=php8.1-*`, then copy the built `.so` into `ini_get('extension_dir')` and verify with `php -m`. Same fix owed to the pgsql sibling hook.
 - **A SAML SSO return-leg "connection timeout" is a missing HTTPS listener, not code. (CONFIRMED + RESOLVED.)** When the gateway's return redirect to `https://<host>/sso` times out in the browser, verify the **EB/ALB HTTPS (443) listener + ACM cert cover the EXACT return hostname before debugging PHP/SAML.** Root cause for Tools was **confirmed**: the `tools.togatech.com` EB environment had **no HTTPS (443) listener configured at all**. Adding the HTTPS:443 listener (with an ACM cert covering the exact return hostname) fixed the SSO return leg completely — **no code change was involved.** Diagnosis signature: `curl` to the public `https://` URL **from the instance itself** returns `http=000` after the full timeout, while `curl http://127.0.0.1` with a `Host:` header returns `200` fast. Reusable for any TOGA app behind Elastic Beanstalk.
 
+- **Likely open — s3fs mount: `006` builds s3fs 1.84, which cannot read the instance role over IMDSv2** (AL2023 EB sets `DisableIMDSv1=true`), so the mount probably fails here. Fix used in api: s3fs 1.97 + fuse3, mounted from a `.platform` predeploy hook — see the [api deploy doc](../../api/workflows/deploy-to-elastic-beanstalk-al2023.md). Not yet verified on Tools.
+
 ## Related
 - [Tools architecture](../architecture.md)
+- [API 1.0 deploy (AL2023, Apache proxy, s3fs 1.97)](../../api/workflows/deploy-to-elastic-beanstalk-al2023.md)
 - [Walmart Activate deploy (same AL2023 pattern, Apache proxy)](../../walmart-activate/workflows/deploy-to-elastic-beanstalk-al2023.md)
 
 ## Change history
+- 2026-10-22 — Correction: nginx-only claim holds only for the nginx proxy; s3fs 1.84 IMDSv2 risk noted (TRUE-83054) (jcardinal)
 - 2026-10-22 — Staging dir corrected `/var/app/ondeck` → `/var/app/staging`; resources now `/opt/resources` + postdeploy symlink (TRUE-83059) (rgirish)

package/knowledge/INDEX.md

@@ -18,7 +18,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **store** (Store) — 4 doc(s) → [1.0/apps/store/INDEX.md](1.0/apps/store/INDEX.md)
 - **tools** (Tools) — 25 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
 - **vision** (Vision) — 0 doc(s) → [1.0/apps/vision/INDEX.md](1.0/apps/vision/INDEX.md)
-- **api** (API 1.0) — 1 doc(s) → [1.0/apps/api/INDEX.md](1.0/apps/api/INDEX.md)
+- **api** (API 1.0) — 2 doc(s) → [1.0/apps/api/INDEX.md](1.0/apps/api/INDEX.md)
 
 ## 2.0 framework
 
@@ -46,7 +46,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **talos-backend** (TOGa Talos) — 33 doc(s) → [standalone/apps/talos-backend/INDEX.md](standalone/apps/talos-backend/INDEX.md)
 - **voice-to-voice** (TOGa Voice) — 6 doc(s) → [standalone/apps/voice-to-voice/INDEX.md](standalone/apps/voice-to-voice/INDEX.md)
 - **ai-bdr** (AI-BDR) — 5 doc(s) → [standalone/apps/ai-bdr/INDEX.md](standalone/apps/ai-bdr/INDEX.md)
-- **forward** (Forwarder) — 3 doc(s) → [standalone/apps/forward/INDEX.md](standalone/apps/forward/INDEX.md)
+- **forward** (Forwarder) — 4 doc(s) → [standalone/apps/forward/INDEX.md](standalone/apps/forward/INDEX.md)
 - **bdr** (BDR) — 9 doc(s) → [standalone/apps/bdr/INDEX.md](standalone/apps/bdr/INDEX.md)
 - **claude** (Claude Harness) — 10 doc(s) → [standalone/apps/claude/INDEX.md](standalone/apps/claude/INDEX.md)
 - **powerbi** (Power BI) — 0 doc(s) → [standalone/apps/powerbi/INDEX.md](standalone/apps/powerbi/INDEX.md)

package/knowledge/clients/nycdoe/features/servicenow-integration.md

@@ -6,7 +6,7 @@ project: Worker
 client: nycdoe
 type: client-feature
 status: active
-updated: 2026-10-19
+updated: 2026-10-22
 owners: [mhammontree, sking, bala]
 files:
   - worker/crons/sync/nycdoe/import_asn.php
@@ -249,6 +249,12 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
     Note this only confirms the row is an eligible *candidate* for the `*/5 min` Stage 5 cron (which then picks ONE ASN at random via `ORDER BY RAND() LIMIT 1` across all pending ASNs) — it does not by itself confirm a ticket was created; wait for the cron and check `togadeskRepairOrderId`/`dtInstallTicketStatusChanged` to confirm ticketing completed.
   - Unlike instances 1–2 below, no duplicate/replacement SO was involved here — the mismatch is a same-ASN sibling PO the warehouse had already been using for other lines on this ASN.
 
+- **⚠ OPEN BUG (TRUE-83064, 2026-10-22; data repaired, root-cause code fix NOT done) — one customer PO split into two ASNs (two vendors) → wrong manufacturer + stranded cross-received units.** Customer PO `WR270089579` came in as two ASNs, one under the Acer vendor and one under the Lexmark vendor, for the same Acer part `DOE-NX.JHWAA.004-NYCDOE` (likely the Acer-as-Lexmark SFTP path, Stage 1b above).
+  - **Manufacturer comes from the ASN vendor, not the part.** Stage 5 (`3_create_installation_ticket.php`) sets the TOGa Desk manufacturer from the ASN's `Vendors.name` and finds the MSO by `advanceShippingNoticeId`. Two ASNs = two MSOs, and the Lexmark-ASN line was ticketed as Lexmark.
+  - **Cross-received serials dead-end silently.** Stage 5 matches an item's units only against receipts of that item's **own** SO. The warehouse received serials onto the **other** PO of the same customer PO, so they never match, and the customer-PO-scoped duplicate-serial guard refuses to add them as new units. No error; the cron retries forever. Evidence: legacy `Logs.API` `getItemReceipt` rows with `sourceJob LIKE '%3_create_installation_ticket%'` (30-day retention).
+  - **Repair (data-only, dbchanges#270, branch `fix/repair-wr270089579-consolidate`):** `Core/SK/2026-10-22-data-repair-wr270089579-consolidate.sql` swaps the 102+102 cross-received `AdvanceShippingNoticeUnits` onto the item whose SO received them and moves item `55578` onto ASN `27180`; `TOGaDeskSupport/SK/2026-10-22-data-repair-wr270089579-consolidate.sql` moves the 24 Lexmark ROs onto MSO `334826` as Acer (assets → manufacturer `10` / model `23668`) and deletes the empty MSO `334820`. Two databases — verify each. Guard rules: [dbchanges workflow](../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md) → *Rules for a one-time data repair*.
+  - **⚠ OUTSTANDING:** TOGa Desk Lexmark model `23782` still holds **152 mislabeled assets** on other POs (`WR270055266` MSO `332014`, `S202652963` MSO `332217`) — not fixed. Any customer PO split across vendors can repeat both failures until Stage 5 is fixed.
+
 - **⚠ OPEN BUG (investigated 2026-06-19, not yet fixed) — manufacturer PO-suffix format variance causes DUPLICATE fulfillment.** Stage 2's serialized dedupe is scoped to the *exact* PO string (`lookupAsnUnitIdByAsnPartNumberAndSerialNumber[vendorId][purchaseOrder][part][serial]`, `legacy_process_asn_queue.php:272`), and the PO flows **verbatim** end to end — never trimmed/normalized except Lenovo's `trim()` in `asnprocessor/lenovo.php`. So when a manufacturer re-sends the *same physical unit* under a different suffix spelling (e.g. Lenovo `WR260150152-RPL` then `WR260150152-RPLE`, same serial `V60095WK`), the two PO strings differ → the serial guard misses → a full duplicate ASN→SO→MSO→RO chain is created for one device (two NetSuite SOs, two repair orders, two onsite tickets).
   - **Confirmed live case:** PO `WR260150152`, serial `V60095WK`, asset `DOE-LN1265478` → repair orders **772308** (MSO 309909, via `-RPL`) and **801064** (MSO 318945, via `-RPLE`), both `ORDER_ASSIGNED_AWAITING_SCHEDULING` (uncaught as of investigation), same TOGa Desk `serialNumId 592519`.
   - **Systemic, not a one-off:** ~14 same-base/same-serial duplicate pairs across **Lenovo AND Lexmark**, spanning suffix forms `-RPL`/`-REPL`/`-RPLE`/`-N`/`_N`. A genuine replacement carries a *new* serial, so identical-serial-across-base-and-suffix = true duplication.
@@ -288,6 +294,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 - Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
 
 ## Change history
+- 2026-10-22 — TRUE-83064: repaired customer PO `WR270089579` (two ASNs, Acer + Lexmark vendor) via dbchanges#270; open-bug gotcha — Stage 5 takes the manufacturer from the ASN vendor and keys the MSO on the ASN; cross-received serials dead-end silently; 152 mislabeled assets on Lexmark model `23782` remain. No code change. (sking)
 - 2026-10-19 — Gated delete live since ~10-11 and its re-release login storm (worker2 SFTP IP ban); file names matter to 1.0 only for FileLog/dedupeKey (rename = re-email); open Lexmark 1969-12-31 date bug. (bala)
 - 2026-10-16 — TRUE-82776: link to the update-rejection monitor; parked RITM1320328 repair-order loop and POD blank-ticket 400s. (mhammontree)
 - 2026-10-14 — TRUE-82735: RITM + POD outbound crons revived (per-MSO loop, own-RITM match, state ownership, POD ordering/isolation), library RITM filter + location lookup fixed, backlog stamps, months-long outage gotcha. Not deployed. (mhammontree)

package/knowledge/clients/office-depot/profile.md

@@ -9,6 +9,7 @@ apps:
   - dbchanges
   - worker2
   - dbchanges2
+  - forward
 project: TOGa Voice
 client: office-depot
 type: profile

package/knowledge/clients/true/profile.md

@@ -18,6 +18,7 @@ apps:
   - api
   - dbchanges
   - toga25-desk
+  - forward
   - toga25-blox
   - toga25-iq
   - apiproxy

package/knowledge/standalone/apps/forward/INDEX.md

@@ -4,4 +4,5 @@
 |-----|---------|
 | [Forwarder Architecture](architecture.md) | Tiny standalone PHP redirect app for TOGA's branded short domains (Apache RewriteMap + light `index.php`); open for the two redirect layers, deploy, decrypt sch |
 | [Encrypted-Link Handler](features/encrypted-link-handler.md) | Forwarder `index.php` handler for redirect domains whose URL path carries an encrypted token (PCMatic/Newegg/`forward.*` agent downloads); open for config + dec |
+| [Forwarder Hosting, DNS and TLS](features/hosting-dns-tls.md) | Where Forwarder runs (EB `forward-production`, prod us-east-1 — the only live env since TRUE-83057), which hostnames point at it per DNS zone, the HTTPS cert, W |
 | [Static Demo Hosting](features/static-demo-hosting.md) | How `demo.togatech.com/<project>/vN/` static demos are hosted: S3 `togatech-demos` + CloudFront (prod account) since 2026-10-09, no longer `forward`; open for t |

package/knowledge/standalone/apps/forward/architecture.md

@@ -6,7 +6,7 @@ project: Forwarder
 client: shared
 type: architecture
 status: active
-updated: 2026-10-09
+updated: 2026-10-22
 owners: [jcardinal]
 files:
   - forward/forward.ini
@@ -15,7 +15,8 @@ files:
   - forward/.platform/httpd/conf.d/rewritemap.conf
   - forward/.ebextensions/rewritemap.config
   - forward/composer.json
-related: []
+related:
+  - standalone/apps/forward/features/hosting-dns-tls.md
 ---
 
 Tiny standalone PHP redirect app for TOGA's branded short domains (Apache RewriteMap + light `index.php`); open for the two redirect layers, deploy, decrypt scheme.
@@ -59,13 +60,13 @@ The `.htaccess` at the repo root drives a plain-text lookup table:
 
 ## Deployment (Elastic Beanstalk / Apache)
 
-- Runs on **Elastic Beanstalk**, Amazon Linux 2023, Apache `httpd`.
+- Runs on **Elastic Beanstalk** env `forward-production` (prod account, us-east-1), Amazon Linux 2023, Apache 2.4.69. DNS, cert, WAF: [hosting-dns-tls](features/hosting-dns-tls.md).
 - The `RewriteMap` definition is deployed to `/etc/httpd/conf.d/rewritemap.conf` two ways for cross-platform safety:
   - `.platform/httpd/conf.d/rewritemap.conf` — the **Amazon Linux 2+/2023** mechanism (current).
   - `.ebextensions/rewritemap.config` — legacy **Amazon Linux 1** `files:` fallback. Both can coexist; AL2+ ignores the `.ebextensions` file path managed by `.platform`, and AL1 ignores `.platform` entirely.
 - `forward.ini` deploys to `/var/www/html/forward.ini` (the path the map points at).
 - `composer.json` declares a `Togatech\Forward\` PSR-4 autoload over `src/` but has **no dependencies** and no `src/` is shipped today — the app is effectively a single `index.php` plus the Apache config.
-- **Three EB environments** run this repo: `agilant-forward-prod` and `true-forward-prod` (legacy account, CodePipeline `Forward-Production`), and `forward-production` (prod account, us-east-1, serves `togatechnology.com`). A change to `forward.ini`/`index.php` must reach all three.
+- **One live EB environment: `forward-production`** (prod, us-east-1); all DNS points there since 2026-10-22 (TRUE-83057). Legacy `agilant-forward-prod` / `true-forward-prod` (legacy account, us-west-2) and their CodePipeline `Forward-Production` are being retired — a deploy through that pipeline does not reach live.
 - **`DirectoryIndex index.html index.php`** in `.htaccess` lets nested static-demo version paths `/<project>/<vN>/` serve via Apache's normal directory index. The single-segment static-demo rewrite rule does not match two-segment paths, and the `index.php` map-miss fallback does not fire for real directories — so versioned demo URLs resolve as directory-index lookups.
 
 ## The `decrypt()` scheme
@@ -84,6 +85,7 @@ Treat this as obfuscation only. Anything genuinely sensitive must not rely on it
 - **No restart needed for new redirects** — edit `forward.ini`, reload httpd. The single most useful operational fact about this app.
 - Specific-path map lines must precede host-only lines (see `forward.ini` header).
 - New **encrypted** domains require a code change (`$encryptedDomains` in `index.php`), not just a `forward.ini` edit.
+- A `forward.ini` destination containing `?` needs the `UnsafeAllow3F` flag on Apache 2.4.60+ (else 403 `AH10508`); kept behind `<IfVersion>` in `.htaccess`. See [hosting-dns-tls](features/hosting-dns-tls.md).
 - `index.php` only runs on a map miss, so a bad/over-broad `forward.ini` host-only line can shadow paths you expected PHP to handle.
 - **EB rejects bundles over 500 MB** ("The artifact is empty or exceeds the maximum size supported by AWS Elastic Beanstalk"), and the CodePipeline EB deploy ignores `.ebignore`. Never commit large static assets here — that broke every deploy from 2026-10-08 until demos moved to S3.
 - **Git history was rewritten on 2026-10-09** (`git filter-repo`) to drop ~500 MB of old demo blobs; a clone is now ~39 MB. Any clone made before that date must be deleted and re-cloned — pulling into it brings the old history back. GitHub's `refs/pull/1/head` and `refs/pull/2/head` still point at old commits (GitHub blocks pushes to them).

package/knowledge/standalone/apps/forward/features/hosting-dns-tls.md

@@ -0,0 +1,56 @@
+---
+title: Forwarder Hosting, DNS and TLS
+framework: "standalone"
+repo: forward
+project: Forwarder
+client: shared
+type: feature
+status: active
+updated: 2026-10-22
+owners: [jcardinal]
+files:
+  - forward/.htaccess
+  - forward/forward.ini
+related:
+  - standalone/apps/forward/architecture.md
+  - standalone/apps/forward/features/static-demo-hosting.md
+---
+
+Where Forwarder runs (EB `forward-production`, prod us-east-1 — the only live env since TRUE-83057), which hostnames point at it per DNS zone, the HTTPS cert, WAF, and the Apache 2.4.60+ `UnsafeAllow3F` fix; open before any DNS, cert, env, or `forward.ini` "?" change.
+
+## Summary
+
+All Forwarder DNS points at EB env **`forward-production`** (PROD account 654654170868, us-east-1, Amazon Linux 2023, Apache 2.4.69). The legacy envs `agilant-forward-prod` and `true-forward-prod` (LEGACY account 502614707982, us-west-2, AL 2018.03 / PHP 7.3) are being shut down for a security issue. Legacy CodePipeline `Forward-Production` (us-west-2) deploys only to those legacy envs and must be disabled/removed when they go. Redirect logic itself: [architecture](../architecture.md).
+
+## How it works
+
+**Hostnames on forward, by DNS zone** (alias target `forward-production.us-east-1.elasticbeanstalk.com`, EB us-east-1 hosted zone `Z117KPS5GTRQ2G`):
+
+| Zone (where) | Names | Record |
+|---|---|---|
+| `togatech.com` (Route 53, PROD) | `aws.devteam`, `devteam`, `feedback`, `itsecurity`, `operations`, `powerbi`, `sos`, `status` | A ALIAS |
+| `togatech.com` (Route 53, PROD) | `customerconnect` | CNAME |
+| `agilantsolutions.com` (Route 53, LEGACY — zone stays there) | apex, `www`, `odp.agent`, `pcmatic.agent`, `sos`, `toga`, `toga.alpha`, `toga.beta`, `togadesk.beta` | cross-account A ALIAS |
+| `goagilant.com` (DNS Made Easy) | apex, `www` | ANAME |
+
+- Cross-account A ALIAS (legacy zone → prod EB) works; create it with the AWS CLI.
+- `asisystem.com` (DNS Made Easy) has no forward records.
+- Deleted 2026-10-22 as dead (404 everywhere, not in `forward.ini` or `$encryptedDomains`): `newegg.agent.togahub.com`, `newegg.agent.truenetworksolutions.com`.
+
+**HTTPS:** set by EB option `aws:elbv2:listener:443` `SSLCertificateArns` (not in the repo). One ACM cert (PROD, us-east-1) covers 10 names (ACM default limit): `togatech.com`, `*.togatech.com`, `togatechnology.com`, `*.togatechnology.com`, `agilantsolutions.com`, `*.agilantsolutions.com`, `*.agent.agilantsolutions.com`, `goagilant.com`, `*.goagilant.com`, `*.devteam.togatech.com`. Chosen over an extra SNI cert via `add-listener-certificates`, which EB does not track (lost on env rebuild/config change).
+
+**WAF / network:** `forward-production` ALB uses WAF `TOGA-Prod` (default Allow, all rules Count); SG open 80/443 to `0.0.0.0/0`. For reference, old `true-forward-prod` used `AgilantWAF` (default Block, IP allowlist + geo); old `agilant-forward-prod` used `TOGA-Legacy` (Count only).
+
+**Apache 2.4.60+ and "?" in a destination:** the CVE-2024-38474 fix makes Apache return `403` (error log `AH10508: Unsafe URL with %3f URL rewritten without UnsafeAllow3F`) when a `forward.ini` destination has a `?` (e.g. `powerbi.togatech.com/sales*` → `app.powerbi.com/links/...?ctid=...`). `.htaccess` fix: both RewriteMap redirect rules carry `UnsafeAllow3F` inside `<IfVersion >= 2.4.60>`; the old rule stays inside `<IfVersion < 2.4.60>` (older Apache rejects the flag). `mod_version` is loaded on both platforms (agilantsolutions/forward#4).
+
+## Gotchas
+
+- **Testing a move/env change: check every `host/path` key in `forward.ini`, not just `/`.** `operations`, `powerbi`, `status` return 404 at `/` but have live path redirects — not dead. The `UnsafeAllow3F` 403 only showed on a path key.
+- Keep `UnsafeAllow3F` inside the `<IfVersion>` guard; unguarded it breaks Apache < 2.4.60.
+- **Cert auto-renewal needs the ACM DNS validation CNAMEs to stay** in the LEGACY `agilantsolutions.com` Route 53 zone and in DNS Made Easy `goagilant.com`. Do not delete them in cleanups.
+- `toga.alpha`, `toga.beta`, `togadesk.beta.agilantsolutions.com` have **no TLS** — two-level names are not covered by `*.agilantsolutions.com` (same as before the move).
+- A new hostname outside the 10 cert names needs a cert change; the cert is at the ACM limit (request a quota raise or a new cert set in `SSLCertificateArns`).
+- Deploys to the legacy pipeline do not reach `forward-production`; do not rely on `Forward-Production` for live changes.
+
+## Change history
+- 2026-10-22 — All DNS moved to `forward-production` (prod us-east-1); legacy envs + pipeline being retired; one ACM cert via `SSLCertificateArns`; `.htaccess` `UnsafeAllow3F` fix for Apache 2.4.60+; two dead newegg names deleted (TRUE-83057) (jcardinal)

package/knowledge/standalone/apps/forward/features/static-demo-hosting.md

@@ -6,7 +6,7 @@ project: Forwarder
 client: shared
 type: feature
 status: active
-updated: 2026-10-09
+updated: 2026-10-22
 owners: ["jcardinal"]
 files:
   - forward/.htaccess
@@ -36,10 +36,10 @@ All in the **PROD** account (654654170868).
 - **Migration (done):** 24 projects (folders with `project.json`) + `tabs.json` = 153 files / 728 MB copied with the same keys, checked byte-for-byte (sha256) through CloudFront, then DNS cutover, then removed from `forward`.
 
 ### Rollback demo.togatech.com to forward EB
-Only works if the demo files are put back in `forward` and it deploys under 500 MB.
+Only works if the demo files are put back in `forward` and it deploys under 500 MB. Target is `forward-production` (the legacy `agilant-forward-prod` env and `Forward-Production` pipeline are retired — see [hosting](hosting-dns-tls.md)).
 1. Restore the demo folders + `tabs.json` into `forward` (copy from S3 or from git history) and confirm the EB bundle is under 500 MB.
-2. Deploy `forward` via `Forward-Production` and confirm it succeeds.
-3. In Route 53 zone `togatech.com` (prod), reverse the change batch: delete the A + AAAA alias records for `demo.togatech.com`, create the CNAME → `agilant-forward-prod` EB host.
+2. Deploy `forward` to EB `forward-production` (prod, us-east-1) and confirm it succeeds.
+3. In Route 53 zone `togatech.com` (prod), delete the A + AAAA alias records for `demo.togatech.com`, create an A ALIAS → `forward-production.us-east-1.elasticbeanstalk.com` (EB zone `Z117KPS5GTRQ2G`).
 4. Confirm `https://demo.togatech.com/<project>` loads.
 
 ### Exporting a Claude Design page
@@ -61,5 +61,6 @@ Before hosting, check: no `<base>` tag, no root-relative/relative asset paths, n
 - [Forwarder Architecture](../architecture.md)
 
 ## Change history
+- 2026-10-22 — Rollback target changed to `forward-production`; legacy forward envs retired (TRUE-83057) (jcardinal)
 - 2026-10-09 — Hosting moved from `forward` EB to S3 `togatech-demos` + CloudFront (prod); DNS cut over; demo folders removed from `forward` (TRUE-82616) (jcardinal)
 - Earlier — demos served from `forward/<name>/index.html` via the `.htaccess` rule, versioned `/vN/` added by the admin tool (jcardinal)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1109",
+  "version": "1.0.1111",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",