toga-ai 1.0.1194 → 1.0.1195

package/knowledge/1.0/apps/worker1.5/INDEX.md

@@ -5,5 +5,6 @@
 | [Worker 1.5 (1.0 Framework) Architecture](architecture.md) | The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for how the crontab is built from schedules/*.json, the overlap check, and what the cr |
 | [worker1.5 — Committed Secrets & Credentials in Logs (debt register)](features/committed-secrets-and-log-exposure.md) | Open security debt in worker1.5: a GitHub token committed in `ebs/git.json` and a cron that writes OptimumDesk passwords into the cron log; open before touching |
 | [worker1.5 cron overlap guard (isProcessRunning, crontab shell, manual runs)](features/cron-overlap-guard.md) | How the 1.0 "already running" check decides a cron exits, why the crontab must use /bin/sh, and how to run a cron by hand on worker15 without it exiting; open w |
+| [Database backups (database_backup_infrastructure.php) — legacy asidbarchives bucket and the new locked prod bucket](features/database-backups.md) | Where prod database backups are written (legacy asidbarchives vs new locked bucket togatech-database-backup-production), how the backup cron thins files, the pr |
 | [worker1.5 PHP ini overrides on EB AL2023 (memory_limit 11G, zz-application.ini, cron log_errors=0)](features/php-ini-memory-limit-eb.md) | How worker1.5 sets PHP ini values (memory_limit 11G, cron `log_errors=0`) on EB `worker15-production`; open before changing any PHP ini setting or the cron comm |
 | [Diagnosing worker1.5 crons that stopped running (single-server 1.0 tier)](workflows/diagnosing-crons-not-running.md) | Steps to tell "worker1.5 crons are not running" from "a downstream system is failing", find the cause on the box, and recover, plus how to triage failing crons |

package/knowledge/1.0/apps/worker1.5/features/database-backups.md

@@ -0,0 +1,73 @@
+---
+title: Database backups (database_backup_infrastructure.php) — legacy asidbarchives bucket and the new locked prod bucket
+framework: "1.0"
+repo: worker1.5
+project: Worker 1.5
+client: shared
+type: feature
+status: active
+updated: 2026-10-29
+owners: ["rgirish"]
+files:
+  - worker1.5/crons/database/database_backup_infrastructure.php
+  - worker1.5/schedules/cron.worker.json
+related:
+  - ../architecture.md
+  - ../workflows/diagnosing-crons-not-running.md
+  - ../../../../standalone/apps/bastion/features/dbdump.md
+---
+
+Where prod database backups are written (legacy asidbarchives vs new locked bucket togatech-database-backup-production), how the backup cron thins files, the prod Aurora topology, and the open risks (backups OFF since 2026-10-22, 30-day thinning vs 35-day lock); open before touching backups or the bucket.
+
+## Summary
+
+The cron `crons/database/database_backup_infrastructure.php` (worker1.5, moved from `worker`) runs `mysqldump` per database and writes gzip files to an s3fs mount. Today it targets the **legacy** bucket `asidbarchives`. A **new, locked** bucket `togatech-database-backup-production` was built (TRUE-83338) to replace it, but nothing writes to it yet.
+
+**⚠ Backups are OFF since 2026-10-22.** In worker1.5 `_main`, `schedules/cron.worker.json` row "Database Backups" is `"active":0` (set when worker 1.0 crons moved to worker1.5). Newest file in `asidbarchives`: `Core_20261022_142038.sql.gz`.
+
+## How it works
+
+### Legacy path (current cron target)
+
+- Bucket `asidbarchives`, legacy account **502614707982**, **us-west-2**. No versioning, no Object Lock, no Block Public Access, no lifecycle, SSE AES256 only. Anyone with write access can wipe it.
+- Mounted with s3fs at `/var/app/asidbarchives`; cron writes `<Database>/<YYYY-MM>/<DD>/<Database>_<YYYYMMDD>_<HHMMSS>.sql.gz`.
+- The cron deletes old files itself, per-database args: keep all for 30 days, hourly to 90, daily to 360, weekly to 720 days.
+- DB credentials are in the cron file itself (plaintext). Never copy them out; see [committed secrets](committed-secrets-and-log-exposure.md).
+
+### New bucket (TRUE-83338, built 2026-10-29, empty)
+
+| Setting | Value |
+|---|---|
+| Name | `togatech-database-backup-production` |
+| Account / region | **654654170868** (new prod) / **us-east-1** (same region as the Aurora writers) |
+| Versioning | on |
+| Encryption | SSE-KMS, customer-managed key `alias/togatech-database-backup`, Bucket Key on, yearly key rotation on |
+| Public access | all 4 Block Public Access on |
+| Object ownership | BucketOwnerEnforced (ACLs off) |
+| Object Lock | enabled at creation; default retention **GOVERNANCE 35 days** |
+| Tags | `Name`, `Ticket=TRUE-83338`, `Purpose=database-backup` |
+| Not yet built | lifecycle rule, bucket policy, IAM writer role |
+
+Proven: a test object got `aws:kms` + GOVERNANCE lock; delete without bypass failed `Access Denied because object protected by object lock.`; removal needed `--bypass-governance-retention`.
+
+Decisions (rgirish, 2026-10-29):
+- Backup bucket **only in prod** (654654170868). None in non-prod account 975050298201 — non-prod data needs no backups.
+- Name pattern `togatech-<purpose>-production`.
+- Region us-east-1 because the DB writers are there.
+
+### Prod 2.0 database topology (what gets backed up)
+
+- Account 654654170868 has 5 Aurora MySQL **global databases**: `production-core-global`, `production-client-global`, `production-logs-global`, `production-archive-global`, `production-cache-global`.
+- Writer cluster `production-<family>-1` in **us-east-1**; read-only secondaries `-2` in us-west-2 and `-3` in eu-west-1. `writer.core.database.togahub.com` → `production-core-1`.
+- Talos Postgres `production-talos` is also us-east-1.
+- Legacy account 502614707982 still holds old us-west-2 clusters: `prod-cluster`, `production-logs-cluster`, `beta1`, `demo1`, `gamma`, `gamma1`, `hotfix1`, `stage1`.
+- us-west-1 RDS calls are denied by an org SCP.
+
+## Gotchas
+
+- **30-day thinning vs 35-day lock.** Pointed at the new bucket as-is, the cron's deletes of day-30+ detailed files fail (GOVERNANCE blocks deletes before day 35). Before switching, pick one: lock 30 days, change the cron's thinning, or drop cron deletes and let an S3 lifecycle rule clean up.
+- Switching the writer also needs an IAM role + bucket policy for the writer, and KMS `GenerateDataKey`/`Decrypt` on the key.
+- Re-enabling only means setting `"active":1` on that row and deploying worker1.5 — but it still writes to the legacy, unprotected bucket.
+
+## Change history
+- 2026-10-29 — New prod backup bucket `togatech-database-backup-production` built (locked 35 days, KMS); found backups OFF since 2026-10-22 and the thinning-vs-lock conflict (TRUE-83338) (rgirish)

package/knowledge/INDEX.md

@@ -7,7 +7,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **library** (Library) _(framework core)_ — 41 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 42 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
-- **worker1.5** (Worker 1.5) — 8 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
+- **worker1.5** (Worker 1.5) — 9 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 19 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
 - **togaview** (TOGa View) — 9 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
 - **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1194",
+  "version": "1.0.1195",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",