toga-ai 1.0.1124 → 1.0.1126

package/knowledge/1.0/apps/worker1.5/INDEX.md

@@ -3,3 +3,4 @@
 | Doc | Summary |
 |-----|---------|
 | [Worker 1.5 (1.0 Framework) Architecture](architecture.md) | The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for how the crontab is built from schedules/*.json, the overlap check, and what the cr |
+| [worker1.5 PHP ini overrides on EB AL2023 (memory_limit 11G, zz-application.ini)](features/php-ini-memory-limit-eb.md) | How worker1.5 sets PHP ini values (memory_limit 11G) on EB `worker15-production`; open before changing any PHP ini setting or wondering why the EB console value |

package/knowledge/1.0/apps/worker1.5/architecture.md

@@ -7,14 +7,16 @@ client: shared
 type: architecture
 status: active
 updated: 2026-10-23
-owners: [bala]
+owners: [bala, jcardinal]
 files:
   - worker1.5/ebs/cron.worker.php
   - worker1.5/.platform/hooks/postdeploy/020_cron.sh
   - worker1.5/schedules/
   - worker1.5/crons/
   - worker1.5/mvc/get.php
+  - worker1.5/ebs/setup_phpini.php
 related:
+  - features/php-ini-memory-limit-eb.md
   - ../worker/architecture.md
   - ../library/architecture.md
   - ../../standards/backend-php.md
@@ -26,7 +28,7 @@ The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for ho
 
 `worker1.5` is the 1.0 `App_` background-job tier on **one Elastic Beanstalk server, PHP 8.5, AL2023** (TRUE-83078). It is a copy of the legacy `worker` repo with the same folder layout (`crons/<area>/...`, `crons/toga2/<client>/`). The 7 role schedules were merged into one file. Role election, heartbeat, stale-worker termination and per-role Elastic IPs are gone: one server runs every active job. Builds on `library`.
 
-**Critical rules:** the schedule `schedules/cron.<ENVIRONMENT>.json` is the source of truth. Production is `cron.worker.json`. A script not in it never runs. A bad `schedule` (not 5 fields) or a `%` in a command aborts the whole crontab install on deploy. Keep cron's default `/bin/sh`: `App_Framework::isProcessRunning()` only ignores `/bin/sh` parent rows, so another shell breaks the already-running check (TRUE-83121). Any PHP warning exits the request or cron (see the backend-php standard, *Any PHP warning terminates the request*).
+**Critical rules:** the schedule `schedules/cron.<ENVIRONMENT>.json` is the source of truth. Production is `cron.worker.json`. A script not in it never runs. A bad `schedule` (not 5 fields) or a `%` in a command aborts the whole crontab install on deploy. Keep cron's default `/bin/sh`: `App_Framework::isProcessRunning()` only ignores `/bin/sh` parent rows, so another shell breaks the already-running check (TRUE-83121). Any PHP warning exits the request or cron (see the backend-php standard, *Any PHP warning terminates the request*). Host setup uses `.platform/hooks` (prebuild/predeploy/postdeploy), not `.ebextensions`; never apply `worker` (AL1) EB fixes here blindly. PHP ini values come from the repo (`ebs/setup_phpini.php` → `/etc/php.d/zz-application.ini`), not the EB console; memory_limit is 11G per process on a 16 GiB m7g.xlarge, so watch crons running at the same time. See [[php-ini-memory-limit-eb]].
 
 ## How a job becomes a cron
 

package/knowledge/1.0/apps/worker1.5/features/php-ini-memory-limit-eb.md

@@ -0,0 +1,37 @@
+---
+title: worker1.5 PHP ini overrides on EB AL2023 (memory_limit 11G, zz-application.ini)
+framework: "1.0"
+repo: worker1.5
+project: Worker 1.5
+client: shared
+type: feature
+status: active
+updated: 2026-10-23
+owners: ["jcardinal"]
+files:
+  - worker1.5/ebs/setup_phpini.php
+  - worker1.5/.platform/hooks/prebuild/030_php_ini.sh
+related: ["../architecture.md", "oneuptime-server-monitor-host-hygiene"]
+---
+
+How worker1.5 sets PHP ini values (memory_limit 11G) on EB `worker15-production`; open before changing any PHP ini setting or wondering why the EB console value is ignored.
+
+## Summary
+
+worker1.5 (separate repo from 1.0 `worker`) runs on EB `worker15-production` (app "Worker", prod account, Amazon Linux 2023 PHP 8.5, m7g.xlarge, 1 instance). PHP `memory_limit` is **11G** for all PHP (CLI crons + php-fpm), set from the repo, not the EB console.
+
+## How it works
+
+- `.platform/hooks/prebuild/030_php_ini.sh` runs `ebs/setup_phpini.php`, which writes `/etc/php.d/zz-application.ini` and deletes the old `/etc/php.d/application.ini`.
+- worker1.5 uses `.platform/hooks` (prebuild/predeploy/postdeploy), **not** `.ebextensions`.
+- php-fpm pool has no `memory_limit` override, so the ini value applies to web and CLI.
+- Verify on the box (SSM): `php -i | grep memory_limit` → 11G.
+
+## Gotchas
+
+- **EB console phpini options go to `/etc/php.d/aws.ini`** (`aws:elasticbeanstalk:container:php:phpini`). Console dropdown caps `memory_limit` at 1G; console/API option values beat `.ebextensions` option_settings.
+- **PHP loads `/etc/php.d/*.ini` in name order; last value wins.** Digit names (`10-`, `20-`, `99-`) sort **before** `aws.ini`, so they lose. App ini must sort after `aws.ini` → `zz-` prefix. Live box had `aws.ini` = 3G; `zz-application.ini` 11G wins.
+- **Capacity risk:** 11G is per process on a 16 GiB box. Several heavy crons at once can OOM the instance.
+
+## Change history
+- 2026-10-23 — memory_limit raised to 11G via repo-tracked zz-application.ini so it wins over console aws.ini (commit 4b07176) (jcardinal)

package/knowledge/INDEX.md

@@ -7,7 +7,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **library** (Library) _(framework core)_ — 41 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 43 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
-- **worker1.5** (Worker 1.5) — 1 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
+- **worker1.5** (Worker 1.5) — 2 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 18 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
 - **togaview** (TOGa View) — 9 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
 - **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1124",
+  "version": "1.0.1126",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",

package/knowledge/sessions/2026-10-22-TRUE-82794-amadda1.md

@@ -1,66 +0,0 @@
----
-type: session
-slug: TRUE-82794
-title: System generated errors not de-escalating (neglect HIGH stuck)
-author: amadda1
-repos: [worker2]
-framework: "2.0"
-client: shared
-status: active
-created: 2026-10-22
-updated: 2026-10-22
----
-
-# Session: TRUE-82794
-**Date:** 2026-10-22
-**Project/Repo:** worker2 (2.0)
-**Task:** Fix the error escalation cron (worker2 `Worker/Infrastructure/Errors.php`, `_Worker_Infrastructure_Errors::Escalate`) so quiet error issues stop sitting at HIGH forever in Logs.Issue and ClickUp.
-
----
-
-## What WORKED
-- **Root cause found and proven in prod Logs DB.** `resolveNeglectUrgency()` raises an issue with an open, unacknowledged ClickUp task to HIGH after 48h and never checks whether the error still fires. Issues with fewer than 3 baseline samples never auto-resolve (`hasDecayedToResolved` needs `baselineSampleCount >= 3`). Prod query 2026-10-22: 329 issues with urgency HIGH, minimumUrgency LOW, open task, `dtAcknowledged IS NULL`, `dtLastOccurred` older than 30 days (oldest 2026-08-03; example `2J` id 36, task 868kkrybk).
-- **Second cause found (CTO review, confirmed in prod).** `loadWorkingSet()` takes `ORDER BY dtLastOccurred DESC LIMIT MAX_ISSUES_PER_RUN` (250). Prod working set = 684, so the oldest ~430 (all the stuck ones) were never evaluated. Every recent `Infrastructure/Errors/Escalate` WorkerJobs row said `"truncated": "working set hit MAX_ISSUES_PER_RUN (250)"`, runs took 2-7s.
-- **Fix committed `c5b56a5`** on `TRUE-82794`, pushed; also merged into `_sandbox-dev` (merge `54acd53`, pushed):
-  - `NEGLECT_QUIET_DAYS = 30`: neglect returns LOW once `dtLastOccurred` is 30+ days old (empty date keeps old behavior).
-  - `MAX_ISSUES_PER_RUN` 250 -> 1000; truncated message reworded (old text "roll to the next run" was false).
-  - `MAX_CLICKUP_SYNCS_PER_RUN = 50`: after 50 syncs, quiet issues (`currentEventCount === 0`) are deferred untouched to next run; firing issues never deferred; `deferred` count in summary. Needed because each sync = up to 3 ClickUp calls with 120ms delay (~1-1.5s), 329 at once would pass the 300s job limit.
-  - Neglect digest (`loadNeglectedIssues`) skips 30-day quiet issues; auto-resolve now counts 1 sync against the cap (both from php-reviewer).
-- **Evidence:** scratch CLI test (stubs + reflection on `resolveNeglectUrgency`, 8 cases) FAILED on old code ("quiet 31 days ... got HIGH"), ALL PASS after. `php -l` clean. php-reviewer: 0 critical, 2 medium fixed.
-- **Pre-deploy prod prediction (read-only):** of the 430 unreached issues, 0 never-notified (no first-notice emails), 0 stale counts (no extra ClickUp writes), 2 will auto-resolve and close their ClickUp tasks under the existing rule.
-
-## What did NOT work — DO NOT RETRY THESE
-- **Only the 30-day check, without raising the limit:** does nothing for the stuck issues, because they are past position 250 and the cron never loads them. The cron does not rotate; every run takes the same newest 250.
-- **Reordering HIGH first (keep 250):** rejected — 580 HIGH issues > 250, oldest HIGH still past the limit.
-- **One-off SQL UPDATE of urgency to LOW:** ClickUp priority would stay High (cron sees "nothing moved" at the early return in `processIssue` and never pushes), and new issues keep getting stuck.
-- **Testing on dev-sandbox:** `error_escalation_enabled` is absent in `Config/sandbox-dev.ini` (= off), so `Escalate()` stops before the changed code. Sandbox Logs has 48 issues, all LOW, 0 IssueClickupTask, 0 IssueEmailAddress. Developer decided to test in prod instead.
-- **Codeception suite locally:** worker2 has no `vendor/` on this machine and the suite needs a local dev ENVIRONMENT; did not run.
-
-## Not tried yet (candidates for next session)
-- `qa-engineer` check (team rule before "done") — developer to decide run or skip.
-- Committed regression test in `tests/Integration/` (can't run locally without vendor/).
-- Tools `$describeNeglect` label (tools/mvc/errors/get.php:252) still shows "neglect HIGH · Nd" next to LOW — developer said don't do it in this ticket.
-- Follow-up ticket: auto-close tasks of quiet low-sample issues so the working set stops growing (else it passes 1000 again later).
-- Unused test seed file (not run): scratchpad `TRUE-82794-sandbox-neglect-test.sql.txt` — temporary, likely gone tomorrow.
-
-## Current file state
-| File | Status | Notes |
-|------|--------|-------|
-| worker2/Worker/Infrastructure/Errors.php | committed `c5b56a5`, pushed to TRUE-82794 and _sandbox-dev | full fix; not in _production yet |
-
-## Decisions made
-- 30 days quiet before neglect stops counting (developer). Rejected 7 days.
-- No `isManaged` / `issueKey` filter on the rule (developer): minimumUrgency floor already protects business issues; 0 of the 329 affected.
-- Raise limit to 1000 + cap ClickUp syncs at 50/run (developer). Rejected: reorder-only, rotation column (needs dbchanges2), auto-close tasks (closes hundreds, out of scope).
-- Tools label change: not in this ticket (developer).
-- Test in prod after deploy, not sandbox (developer).
-- Wake-up behavior kept: a 30-day quiet issue that fires again jumps straight to HIGH (old untouched task) and emails — same as today.
-
-## Blockers
-none
-
-## Exact next step
-> Run `/start TRUE-82794`, then `/session-resume latest`; decide on the `qa-engineer` check, then run `/finish` to open the PR TRUE-82794 -> _production. After merge + prod deploy, check `Core.WorkerJobs` Escalate rows (issuesChecked ~684, failures 0, deferred -> 0) and that `2J` is LOW in Logs.Issue and Low in ClickUp (868kkrybk).
-
----
-_Saved by /session-save on 2026-10-22_