toga-ai 1.0.1193 → 1.0.1195

package/knowledge/1.0/apps/worker1.5/INDEX.md

@@ -5,5 +5,6 @@
 | [Worker 1.5 (1.0 Framework) Architecture](architecture.md) | The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for how the crontab is built from schedules/*.json, the overlap check, and what the cr |
 | [worker1.5 — Committed Secrets & Credentials in Logs (debt register)](features/committed-secrets-and-log-exposure.md) | Open security debt in worker1.5: a GitHub token committed in `ebs/git.json` and a cron that writes OptimumDesk passwords into the cron log; open before touching |
 | [worker1.5 cron overlap guard (isProcessRunning, crontab shell, manual runs)](features/cron-overlap-guard.md) | How the 1.0 "already running" check decides a cron exits, why the crontab must use /bin/sh, and how to run a cron by hand on worker15 without it exiting; open w |
+| [Database backups (database_backup_infrastructure.php) — legacy asidbarchives bucket and the new locked prod bucket](features/database-backups.md) | Where prod database backups are written (legacy asidbarchives vs new locked bucket togatech-database-backup-production), how the backup cron thins files, the pr |
 | [worker1.5 PHP ini overrides on EB AL2023 (memory_limit 11G, zz-application.ini, cron log_errors=0)](features/php-ini-memory-limit-eb.md) | How worker1.5 sets PHP ini values (memory_limit 11G, cron `log_errors=0`) on EB `worker15-production`; open before changing any PHP ini setting or the cron comm |
 | [Diagnosing worker1.5 crons that stopped running (single-server 1.0 tier)](workflows/diagnosing-crons-not-running.md) | Steps to tell "worker1.5 crons are not running" from "a downstream system is failing", find the cause on the box, and recover, plus how to triage failing crons |

package/knowledge/1.0/apps/worker1.5/features/database-backups.md

@@ -0,0 +1,73 @@
+---
+title: Database backups (database_backup_infrastructure.php) — legacy asidbarchives bucket and the new locked prod bucket
+framework: "1.0"
+repo: worker1.5
+project: Worker 1.5
+client: shared
+type: feature
+status: active
+updated: 2026-10-29
+owners: ["rgirish"]
+files:
+  - worker1.5/crons/database/database_backup_infrastructure.php
+  - worker1.5/schedules/cron.worker.json
+related:
+  - ../architecture.md
+  - ../workflows/diagnosing-crons-not-running.md
+  - ../../../../standalone/apps/bastion/features/dbdump.md
+---
+
+Where prod database backups are written (legacy asidbarchives vs new locked bucket togatech-database-backup-production), how the backup cron thins files, the prod Aurora topology, and the open risks (backups OFF since 2026-10-22, 30-day thinning vs 35-day lock); open before touching backups or the bucket.
+
+## Summary
+
+The cron `crons/database/database_backup_infrastructure.php` (worker1.5, moved from `worker`) runs `mysqldump` per database and writes gzip files to an s3fs mount. Today it targets the **legacy** bucket `asidbarchives`. A **new, locked** bucket `togatech-database-backup-production` was built (TRUE-83338) to replace it, but nothing writes to it yet.
+
+**⚠ Backups are OFF since 2026-10-22.** In worker1.5 `_main`, `schedules/cron.worker.json` row "Database Backups" is `"active":0` (set when worker 1.0 crons moved to worker1.5). Newest file in `asidbarchives`: `Core_20261022_142038.sql.gz`.
+
+## How it works
+
+### Legacy path (current cron target)
+
+- Bucket `asidbarchives`, legacy account **502614707982**, **us-west-2**. No versioning, no Object Lock, no Block Public Access, no lifecycle, SSE AES256 only. Anyone with write access can wipe it.
+- Mounted with s3fs at `/var/app/asidbarchives`; cron writes `<Database>/<YYYY-MM>/<DD>/<Database>_<YYYYMMDD>_<HHMMSS>.sql.gz`.
+- The cron deletes old files itself, per-database args: keep all for 30 days, hourly to 90, daily to 360, weekly to 720 days.
+- DB credentials are in the cron file itself (plaintext). Never copy them out; see [committed secrets](committed-secrets-and-log-exposure.md).
+
+### New bucket (TRUE-83338, built 2026-10-29, empty)
+
+| Setting | Value |
+|---|---|
+| Name | `togatech-database-backup-production` |
+| Account / region | **654654170868** (new prod) / **us-east-1** (same region as the Aurora writers) |
+| Versioning | on |
+| Encryption | SSE-KMS, customer-managed key `alias/togatech-database-backup`, Bucket Key on, yearly key rotation on |
+| Public access | all 4 Block Public Access on |
+| Object ownership | BucketOwnerEnforced (ACLs off) |
+| Object Lock | enabled at creation; default retention **GOVERNANCE 35 days** |
+| Tags | `Name`, `Ticket=TRUE-83338`, `Purpose=database-backup` |
+| Not yet built | lifecycle rule, bucket policy, IAM writer role |
+
+Proven: a test object got `aws:kms` + GOVERNANCE lock; delete without bypass failed `Access Denied because object protected by object lock.`; removal needed `--bypass-governance-retention`.
+
+Decisions (rgirish, 2026-10-29):
+- Backup bucket **only in prod** (654654170868). None in non-prod account 975050298201 — non-prod data needs no backups.
+- Name pattern `togatech-<purpose>-production`.
+- Region us-east-1 because the DB writers are there.
+
+### Prod 2.0 database topology (what gets backed up)
+
+- Account 654654170868 has 5 Aurora MySQL **global databases**: `production-core-global`, `production-client-global`, `production-logs-global`, `production-archive-global`, `production-cache-global`.
+- Writer cluster `production-<family>-1` in **us-east-1**; read-only secondaries `-2` in us-west-2 and `-3` in eu-west-1. `writer.core.database.togahub.com` → `production-core-1`.
+- Talos Postgres `production-talos` is also us-east-1.
+- Legacy account 502614707982 still holds old us-west-2 clusters: `prod-cluster`, `production-logs-cluster`, `beta1`, `demo1`, `gamma`, `gamma1`, `hotfix1`, `stage1`.
+- us-west-1 RDS calls are denied by an org SCP.
+
+## Gotchas
+
+- **30-day thinning vs 35-day lock.** Pointed at the new bucket as-is, the cron's deletes of day-30+ detailed files fail (GOVERNANCE blocks deletes before day 35). Before switching, pick one: lock 30 days, change the cron's thinning, or drop cron deletes and let an S3 lifecycle rule clean up.
+- Switching the writer also needs an IAM role + bucket policy for the writer, and KMS `GenerateDataKey`/`Decrypt` on the key.
+- Re-enabling only means setting `"active":1` on that row and deploying worker1.5 — but it still writes to the legacy, unprotected bucket.
+
+## Change history
+- 2026-10-29 — New prod backup bucket `togatech-database-backup-production` built (locked 35 days, KMS); found backups OFF since 2026-10-22 and the thinning-vs-lock conflict (TRUE-83338) (rgirish)

package/knowledge/INDEX.md

@@ -7,7 +7,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **library** (Library) _(framework core)_ — 41 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 42 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
-- **worker1.5** (Worker 1.5) — 8 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
+- **worker1.5** (Worker 1.5) — 9 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 19 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
 - **togaview** (TOGa View) — 9 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
 - **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)

package/knowledge/clients/nycdoe/features/servicenow-integration.md

@@ -6,7 +6,7 @@ project: Worker
 client: nycdoe
 type: client-feature
 status: active
-updated: 2026-10-27
+updated: 2026-10-29
 owners: [mhammontree, sking, bala, jcardinal, rgirish]
 files:
   - worker1.5/crons/sync/nycdoe/3_create_installation_ticket.php
@@ -132,7 +132,10 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘      [UNIQUE ded
 - **Stage 5:** for units `WHERE togadeskRepairOrderId IS NULL`, reads serials from NetSuite item receipts (so TOGa Desk RO serials are **driven by NetSuite**, not the ASN unit), creates/reuses TOGa Desk manufacturers/models/assets/serial-numbers/MSO and creates **a new repair_orders row unconditionally per loop iteration** — one RO per Unit (a 50-cable line = 1 unit = 1 RO). Existing-serial lookup is scoped **per item row** (`$asnItemId`). Item receipts are resolved **Sales-Order-wide** via `App_NetSuite::listItemReceiptsCreatedFromSalesOrder($soId)` (fans across ALL POs from `listPurchaseOrdersCreatedFromSalesOrder`, dedupes receipts by `internalId`), cached **per sales-order internal id** (`$itemReceiptsBySo`, keyed by `netSuiteInternalSalesOrderId`) — a shared SO is still fetched once — with the `getItemReceipt` detail cache kept per (globally-unique) receipt `internalId`. Both the inner item query **and** the outer ASN-candidate query require `netSuiteInternalSalesOrderId <> 0` (a sentinel guard mirroring the PO guard) so a null/zero-SO row cannot poison the cache or match the wrong SO's receipts. Self-healing: units previously stuck under an unstamped PO clear on subsequent 5-min cron runs.
   - **Receipt resolution is SO-ONLY — the stamped PO id never finds a receipt** (traced 2026-07-29, `3_create_installation_ticket.php`: candidate query ~L248-278, per-item ~L311-341, receipt resolution ~L354-360, part-number match ~L418-422, per-unit gate ~L511-535). The chain is strictly `AdvanceShippingNoticeItems.netSuiteInternalSalesOrderId` → `listPurchaseOrdersCreatedFromSalesOrder` (all POs created from that SO) → `listItemReceiptsCreateFromPurchaseOrder` for each → dedupe (this is what `listItemReceiptsCreatedFromSalesOrder` wraps, `library/app/netsuite.php` ~L739 / ~L762 / ~L797, details via `getItemReceipt` ~L1697). **`netSuiteInternalPurchaseOrderId` is NEVER used to look up receipts** — it is only a non-null / non-zero **eligibility gate**. The schema is misleading here: the PO column *looks* authoritative and is not.
   - **Item receipts are never persisted.** They are fetched **live from NetSuite on every run** (no table, no bridge row), and the receipt-line → ASN-line linkage is derived **at runtime by part-number string match** after stripping the `DOE-` prefix and `_` characters. So NetSuite is the only source of truth for what was received, and a part-number formatting drift breaks the match silently.
-  - **Candidate pick is `ORDER BY RAND() LIMIT 1` ON PURPOSE — do not change it to a fixed order** (rgirish, 2026-10-23). Newest-first was tried and the script stuck for days on one ASN with bad data. Cost: ~1,698 pending candidate ASNs (oldest 2023-10-02, most will never complete) and only a handful finish per run, so a new, fully received ASN can wait hours or days by chance (2026-10-22: 4 ASNs completed in 7.5 h). Symptom "received in NetSuite, no repair order in Desk" with correct SO/PO stamps = this wait, not a stamp mismatch → run the ASN by hand (below).
+  - **Candidate pick is `ORDER BY RAND() LIMIT 1` ON PURPOSE — do not change it to a fixed order** (rgirish, 2026-10-23). Newest-first was tried and the script stuck for days on one ASN with bad data. Cost: ~1,700–1,800 pending candidate ASNs (1,698 on 2026-10-23, 1,795 on 2026-10-29; oldest 2023-10-02, most will never complete) and only a handful finish per run, so a new, fully received ASN can wait hours or days by chance (2026-10-22: 4 ASNs completed in 7.5 h). Symptom "received in NetSuite, no repair order in Desk" with correct SO/PO stamps = this wait, not a stamp mismatch. A DOE report that lists POs by repair order will miss the PO until then.
+    - **Default response for ONE such ASN: no action** — no manual run, no code change; let the cron pick it (Brian Yard, 2026-10-29). Run by hand (below) only when asked, or for a batch/outage gap.
+    - **Fast triage (SuiteQL, 2 queries):** `transaction` + `transactionline` `WHERE id IN (<SO>,<PO>) OR createdfrom IN (<SO>,<PO>)` shows the SO → PO → ItemRcpt chain at once; `inventoryassignment WHERE transaction = <receipt id>` shows the received serial. PO created from SO + receipt holds the ASN serial = clean chain → it is this wait, not [Pre-ASN Manual Receipt Mismatch](#gotchas).
+    - Worked case (TRUE-83394): `WR270047873` = ASN 27376 (Apple, 1 unit, serial `CK7W74WQCG`), late from the TRUE-82940 Stage 1b outage; SO 7686363 → PO 7686364 → receipt 7693049 all clean on 10-28; still no RO on 10-29 → left to the cron.
   - **Manual one-ASN run** (optional argv `<asnId>`, int-cast, non-numeric throws "Invalid ASN id"; pins the candidate query to `AdvanceShippingNoticeItems.advanceShippingNoticeId`; no arg = normal cron): as root on worker15 via SSM, `php /var/app/current/crons/sync/nycdoe/3_create_installation_ticket.php <asnId>` (or `cd` + `./`). Never use the cron's own `/var/www/html/...` path — the overlap guard exits it at once, silently ([path rules](../../../1.0/apps/worker1.5/features/cron-overlap-guard.md)). Run ASNs **one after another**. Verify after: per MSO, `COUNT(repair_orders) = COUNT(DISTINCT serialNumId)`.
   - **MSO reuse is safe on re-run.** `doeCreateInstallationRepairOrder()` looks the managed service order up by `advanceShippingNoticeId`, so re-running Stage 5 against an ASN that already has an MSO **reuses** it rather than creating a duplicate.
   - **The per-SO (not per-ASN) receipt cache is load-bearing.** An ASN can carry lines from more than one NetSuite sales order; caching the receipt set once per ASN off the *first* line's SO matches every other line against the wrong SO's receipts (FIXED 2026-07-22 — see the multi-SO gotcha). One level *above* the multi-PO-per-SO fix.
@@ -315,6 +318,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 - Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
 
 ## Change history
+- 2026-10-29 — TRUE-83394: `WR270047873` received in NetSuite but no Desk RO = Stage 5 random-pick wait (1,795 candidates); default = no action for one ASN (Brian Yard); SuiteQL chain triage tip. No code change. (mhammontree)
 - 2026-10-27 — TRUE-82940: `apple.php:74` PHP 8.5 outage (10-22 → 27) fixed, not deployed; errors land in V2 `Logs.Issue`; Stage 1b cadence is 15 min (worker1.5); gated mode sends one `RunForFiles` per account per run. (mhammontree)
 - 2026-10-23 — Daily open/closed report row; manual re-run gotcha (duplicate email, 24 h window shift) (TRUE-83159) (bala)
 - 2026-10-23 — RITM cron still broken in the worker1.5 copy (undefined vars); not fixed on TRUE-83163 (jcardinal)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1193",
+  "version": "1.0.1195",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",