toga-ai 1.0.1122 → 1.0.1124

package/knowledge/1.0/apps/tools/features/errors-curation-console.md

@@ -6,8 +6,8 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: ["jcardinal", "mhammontree"]
+updated: 2026-10-23
+owners: ["jcardinal", "mhammontree", "amadda"]
 files:
   - tools/mvc/errors/get.php
   - tools/mvc/errors/post.php
@@ -140,7 +140,7 @@ The Core Logs tables this console queries are `Issue`, `Event`, `IssueFingerprin
 ## Gotchas
 - **⚠ Recipients attach to an `issueId`, so they cannot be configured in advance.** A new business exception's recipients can only be added **after** the condition first fires and creates the `Logs.Issue` row — declaring a stable `issueKey` in code does **not** let you pre-wire the email list. Therefore the **first occurrence always routes to a developer ClickUp task**, and somebody has to come back here and add recipients (which is also what flips `isManaged = 1`). Live example: issue **208** `COMPASS_OFFICE_DEPOT_PO_IMPORT_REJECTED_OVER_QUANTITY` is still ClickUp-routed because nobody did. See [error reporting — Issue/Event capture](../../../../2.0/apps/_underscore/features/error-reporting-issue-event.md).
 - **⚠ OPEN BUG (found 2026-08-04, deliberately left unfixed pending the developer's go-ahead): `tools/mvc/errors/post.php` ~line 181, the merge-fingerprint action, will merge into the WRONG issue.** It takes a reference **or** a numeric id in **one** field and disambiguates with `ctype_digit()`. That was safe under the old **base-26** scheme, where every reference contained a letter. Under the new **base-32** alphabet an all-digit reference like `'68'` is treated as **id 68** and silently repoints the fingerprint at the wrong Issue. Fix is either reference-first-then-id precedence or two separate inputs (as `/errors/issue` now does).
-- **The 24/48 neglect thresholds are duplicated as plain numbers in both Tools views.** Tools is 1.0 and cannot read worker2's `NEGLECT_NORMAL_HOURS` / `NEGLECT_HIGH_HOURS`; both sites carry a comment naming those constants as the source of truth. **Changing the policy means changing three places** (worker2 + both Tools views).
+- **The 24/48 neglect thresholds are duplicated as plain numbers in both Tools views.** Tools is 1.0 and cannot read worker2's `NEGLECT_NORMAL_HOURS` / `NEGLECT_HIGH_HOURS`; both sites carry a comment naming those constants as the source of truth. **Changing the policy means changing three places** (worker2 + both Tools views). Known unfixed (2026-10-23): Tools does not apply worker2's `NEGLECT_QUIET_DAYS` (30), so `$describeNeglect` (`tools/mvc/errors/get.php:252`) still shows "neglect HIGH · Nd" beside a LOW urgency on 30-day-quiet issues.
 - **A 2.0 table rename breaks this page silently until it is run.** This console has **no model layer** — `get.php` and `post.php` interpolate table names into hand-written `App_Database` SQL (40+ literal occurrences between them). Nothing here fails at deploy time; it fails when a curator opens the page. Any rename in the shared Core Logs DB must be applied to both files in the same release as the migration.
 - **This is a 1.0 app reading a 2.0 shared database.** Access is plain `App_Database` via the `db_toga2logs` alias, not the `_Model` layer — so none of the 2.0 model protections apply. Escape every interpolated value and allowlist every identifier (see the 1.0 back-end standard).
 - **Cross-tenant surface.** The Core Logs DB is shared by every client, so this console can show one client's captured context to a viewer looking at another client's problem. It is internal-only and persona-gated for that reason — do not expose any part of it to a client.

package/knowledge/2.0/apps/worker2/features/error-escalation-cron.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: ["jcardinal", "dfranks"]
+updated: 2026-10-23
+owners: ["jcardinal", "dfranks", "amadda"]
 files:
   - worker2/Worker/Infrastructure/Errors.php
   - worker2/Worker/Notification/Email.php
@@ -44,7 +44,7 @@ Sole owner of error-pipeline notification (escalation, de-escalation, ClickUp ti
 ### Urgency = max of two independent axes, plus a floor
 
 - **Volume** — occurrences in a **1-hour** window: **NORMAL 5 / HIGH 10 / URGENT 20** (retuned 2026-08-04 from 20/50/100 over 10 minutes — a 10-minute window never fired for the low-and-steady failures that matter).
-- **Neglect** — hours an *unacknowledged* ClickUp task has sat: **NORMAL 24 / HIGH 48. Capped at HIGH — no URGENT rung.**
+- **Neglect** — hours an *unacknowledged* ClickUp task has sat: **NORMAL 24 / HIGH 48. Capped at HIGH — no URGENT rung.** Returns LOW once `dtLastOccurred` is **`NEGLECT_QUIET_DAYS` (30)** days old (see below).
 - **Sustained-24h** — an issue still occurring a day later escalates on duration alone.
 - **Floor** — the Issue's declared `minimumUrgency` applies under *all* axes.
 
@@ -62,6 +62,22 @@ Why: neglect answers *"is anyone on this"* — a question about the team's respo
 
 **Deploy effect.** All 50 then-HIGH issues had open unacknowledged tasks younger than 24h, so their neglect contribution becomes LOW and they recompute from volume/sustained/floor within ~3 runs of the worker2 deploy. Expect a **burst of ClickUp priority PUTs** on the first runs as `clickupPriority` re-syncs.
 
+#### Quiet issues stuck at HIGH — 30-day release + working-set caps (fixed 2026-10-23, TRUE-82794)
+
+Prod 2026-10-22: **329** issues sat at HIGH from neglect alone (oldest 2026-08-03, e.g. 2J, `Logs.Issue` id 36). Two causes:
+
+- **A — neglect ignored silence.** `resolveNeglectUrgency()` raised any open, unacknowledged task to HIGH after 48h whether or not the error still fired. Issues with `baselineSampleCount < 3` never auto-resolve ([auto-resolution](./error-issue-auto-resolution.md)), so their tasks never close and they stayed HIGH forever.
+- **B — the working set never reached them.** `loadWorkingSet()` is `ORDER BY dtLastOccurred DESC LIMIT MAX_ISSUES_PER_RUN` (was 250) with prod at ~680 rows. The cron does **not rotate** — every run takes the same newest rows, so the oldest ~430 were never evaluated. Symptom in every run output: `working set hit MAX_ISSUES_PER_RUN (250)`.
+
+Fix (`Worker/Infrastructure/Errors.php`):
+
+- **`NEGLECT_QUIET_DAYS = 30`** — neglect contributes LOW once `dtLastOccurred` is 30+ days old (empty date keeps old behavior). `minimumUrgency` floor still applies; drop still goes through the `REQUIRED_QUIET_RUNS` (3) dwell. A quiet issue that fires again returns to normal escalation and can jump straight back to HIGH — on purpose.
+- **`MAX_ISSUES_PER_RUN` 250 → 1000.**
+- **`MAX_CLICKUP_SYNCS_PER_RUN = 50`.** After 50 syncs, quiet issues (`currentEventCount === 0`) are **deferred untouched** to the next run; firing issues are never deferred. An auto-resolve counts as 1 sync. Run summary gains a `deferred` count. Why: one sync = up to 3 ClickUp calls with 120ms delay (~1-1.5s); syncing all 329 in one run would pass the 300s job limit.
+- **`loadNeglectedIssues()`** (digest) skips 30-day-quiet issues.
+
+Rejected: 7 days quiet (too short); filtering on `isManaged`/`issueKey` (floor already protects business issues, 0 of 329 affected); ordering HIGH first (580 HIGH > 250, still starves); a rotation column (needs `dbchanges2`); one-off SQL `UPDATE` to LOW (ClickUp priority stays High because `processIssue` early-returns when nothing moved, and new issues keep getting stuck); auto-closing the tasks (closes hundreds, out of scope).
+
 ### Daily neglect digest email (`Infrastructure/Errors/NeglectDigest`)
 
 Second worker action — `public NeglectDigest()`, with `loadNeglectedIssues()` and `buildNeglectDigestBody()` — registered in `Core.CronJobs` as action `Infrastructure/Errors/NeglectDigest` at **`0 8 * * *`** (08:00 CT: after overnight cleanup, before the working day), `maxExecutionTime` 300. Migration: `dbchanges2/Core/2026-08-04a - Error neglect digest cron job.sql`.
@@ -194,7 +210,9 @@ Diagnosed from `Core.WorkerJobs` id 583346 (`Team/Transcripts/Export`). **Any ne
 ## Gotchas
 
 - **⚠ This cron is an EXFILTRATION PATH for `Event.context` — treat anything captured as ticket-visible and email-visible.** As of 2026-08-04 capture writes a `GLOBALS` group into `Event.context`, and redaction is **by key name only**, so value-recognisable PII (names, city, state, ZIP, publishable-looking ids) survives into ClickUp task bodies and escalation emails — neither single-tenant nor access-controlled the way Sentry was. Before widening what this cron copies out of `context`, read the security note in the [2.0 error-reporting doc](../../_underscore/features/error-reporting-issue-event.md). Mitigations are proposed but **not applied**.
-- **An issue held at HIGH purely by neglect never de-escalates on its own.** `applyDeEscalationDwell()` needs the computed target to sit **below** the stored urgency for `REQUIRED_QUIET_RUNS` (3) consecutive runs, but the neglect axis keeps recomputing the target **at** the stored level — so the dwell counter never starts. That is why issue 3H showed `consecutiveQuietWindows = 0` after five hours of total silence. **Acknowledging the ClickUp task releases it**, and it then drops within 3 cron runs (~3 min).
+- **An issue held at HIGH purely by neglect does not de-escalate until 30 days quiet.** `applyDeEscalationDwell()` needs the computed target **below** the stored urgency for `REQUIRED_QUIET_RUNS` (3) runs; the neglect axis keeps the target **at** HIGH, so the dwell never starts (issue 3H: `consecutiveQuietWindows = 0` after hours of silence). Release paths: acknowledge the ClickUp task (drops within ~3 runs), or `NEGLECT_QUIET_DAYS` passes.
+- **`MAX_ISSUES_PER_RUN` is a hard cut, not a page.** Rows past the limit are never evaluated — no rotation. If run output says `working set hit MAX_ISSUES_PER_RUN`, issues are being starved. The set keeps growing (quiet low-sample issues never close their tasks) and may pass 1000 again — known, follow-up candidate.
+- **Cannot be tested on dev-sandbox.** `error_escalation_enabled` is absent from `Config/sandbox-dev.ini` (= off), so `Escalate()` returns before any of this runs; sandbox Logs also has only LOW issues and no `IssueClickupTask` rows. Verify in prod (normal runs: 1.5-6.3s at 250 issues).
 - **Never send email inline inside this cron's open transaction — enqueue it.** `_Email::send()` registers a second connection and toggles the read host, and a throw from in there was only ever `error_log`'d. Use `_Worker::runTask` so a failure becomes a visible failed `WorkerJobs` row.
 - **A "nothing changed, return early" shortcut must not sit above the notification block.** It consumed reminder-ladder rungs without sending anything, for months, while reporting `emailsSent:0 / failures:0` as success. Decide the outbound write and the notification independently.
 - **Persist state before, or independently of, any outbound call.** A regression chain took the whole run down every minute: `syncCustomFields()` read `$issue->dtLastOccurred`, which `loadWorkingSet()` never selected; `_Error::errorHandler()` promotes the undefined-property warning to an **exception**, which killed the run **right after the ClickUp PUT and before `persistIssueState()`** — so urgency never persisted and every run recomputed the same "change". Three fixes: select the missing columns; wrap `processTechnicalIssue()` in try/catch so state **always** persists even when an outbound call fails; and store `Issue.clickupPriority` instead of inferring what was pushed. **A missing column in a working-set SELECT is a runtime exception in 2.0, not a null.**

package/knowledge/sessions/2026-10-23-TRUE-82794-amadda.md

@@ -0,0 +1,74 @@
+---
+type: session
+slug: TRUE-82794
+title: System generated errors not de-escalating (neglect HIGH stuck) — PR open, waiting on review + deploy
+author: amadda
+repos: [worker2]
+framework: "2.0"
+client: shared
+status: active
+created: 2026-10-22
+updated: 2026-10-23
+---
+
+# Session: TRUE-82794
+**Date:** 2026-10-23
+**Project/Repo:** worker2 (2.0)
+**Task:** Fix the error escalation cron (worker2 `Worker/Infrastructure/Errors.php`, `_Worker_Infrastructure_Errors::Escalate`) so quiet error issues stop sitting at HIGH forever in Logs.Issue and ClickUp. Code is done; PR #208 to `_production` is open and waiting on review, merge and prod deploy.
+
+---
+
+## What WORKED
+- **Fix committed `c5b56a5`** on `TRUE-82794` (pushed; also merged into `_sandbox-dev` as `54acd53` on 2026-10-22). Only file: `Worker/Infrastructure/Errors.php` (+37 / -3).
+  - `NEGLECT_QUIET_DAYS = 30`: `resolveNeglectUrgency()` returns LOW once `dtLastOccurred` is 30+ days old (empty date keeps old behavior).
+  - `MAX_ISSUES_PER_RUN` 250 -> 1000.
+  - `MAX_CLICKUP_SYNCS_PER_RUN = 50`: quiet issues (`currentEventCount === 0`) deferred after 50 syncs; firing issues never deferred; `deferred` count in summary; auto-resolve counts 1 sync.
+  - Neglect digest (`loadNeglectedIssues`) skips 30-day quiet issues.
+- **qa-engineer check-only (2026-10-23): PASS.** Scratch harness (outside repo) loaded the real Errors.php with stubs: 35 passed, 0 failed. All 5 claims pass; minimumUrgency floor holds; REQUIRED_QUIET_RUNS (3) still applies. No bugs.
+- **Prod read-only check (2026-10-23, before deploy):** working set 671 (< 1000); 327 issues stuck HIGH (minimumUrgency LOW, open task, unacknowledged, quiet 30+ days); `2J` (Logs.Issue id 36) still HIGH, dtLastOccurred 2026-08-03 17:05:09, task `868kkrybk` open. Current Escalate runs: issuesChecked 250, 1.5–6.3s, failures 0, output says "working set hit MAX_ISSUES_PER_RUN (250)".
+- **/finish done (2026-10-23):**
+  - PR https://github.com/agilantsolutions/worker2/pull/208 (`TRUE-82794` -> `_production`), risk LOW.
+  - KB updated + pushed: `2.0/apps/worker2/features/error-escalation-cron.md`, `1.0/apps/tools/features/errors-curation-console.md`.
+  - Ticket fields: Application ClickUp -> ClickUp + Worker 2.0; Has UI Changes -> unchecked. PM comment posted.
+  - Status set to `development finished`; worker2 automation (`_Worker_Clickup::handleDevelopmentFinished`) moved it to `front-end review` (has PR, no UI changes) — expected.
+- Branch is 4 commits behind `_production`, but those touch only Elite/Compass files — no conflict, no rebase needed.
+
+## What did NOT work — DO NOT RETRY THESE
+- **Only the 30-day check, without raising the limit:** does nothing for the stuck issues, because they are past position 250 and the cron never loads them. The cron does not rotate; every run takes the same newest 250.
+- **Reordering HIGH first (keep 250):** rejected — 580 HIGH issues > 250, oldest HIGH still past the limit.
+- **One-off SQL UPDATE of urgency to LOW:** ClickUp priority would stay High (cron sees "nothing moved" at the early return in `processIssue` and never pushes), and new issues keep getting stuck.
+- **Testing on dev-sandbox:** `error_escalation_enabled` is absent in `Config/sandbox-dev.ini` (= off), so `Escalate()` stops before the changed code. Sandbox Logs has 48 issues, all LOW, 0 IssueClickupTask, 0 IssueEmailAddress.
+- **Codeception suite locally:** worker2 has no `vendor/` on this machine and the suite needs a local dev ENVIRONMENT; did not run.
+- **qa-engineer subagent querying prod:** the TOGa DB MCP tool was not available inside the subagent ("No such tool available"); run prod queries from the main session instead.
+- **ticket-reader `mode: options`:** failed ("options not built ... parser rejects the get_custom_fields output"); got Application option ids with `jq` from the saved `get_custom_fields` output instead (Worker 2.0 = `19c2cbee-6b1d-4f24-8222-89a95d793459`).
+- **Setting the ticket back to Development Finished:** do not — it re-runs the QA merge into the QA branch and lands on `front-end review` again.
+
+## Not tried yet (candidates for next session)
+- Post-deploy prod check (see Exact next step).
+- Committed regression test in `tests/Integration/` (can't run locally without vendor/) — flagged in PR TEAM RULE CHECK.
+- Tools `$describeNeglect` label (tools/mvc/errors/get.php:252) still shows "neglect HIGH · Nd" next to LOW — developer said not in this ticket.
+- Follow-up ticket: auto-close tasks of quiet low-sample issues so the working set stops growing (else it passes 1000 again later).
+
+## Current file state
+| File | Status | Notes |
+|------|--------|-------|
+| worker2/Worker/Infrastructure/Errors.php | committed `c5b56a5`, pushed, in PR #208 | not in `_production` yet |
+
+## Decisions made
+- 30 days quiet before neglect stops counting (developer). Rejected 7 days.
+- No `isManaged` / `issueKey` filter (developer): minimumUrgency floor already protects business issues.
+- Raise limit to 1000 + cap ClickUp syncs at 50/run (developer). Rejected: reorder-only, rotation column (needs dbchanges2), auto-close tasks (out of scope).
+- Tools label change: not in this ticket (developer).
+- Test in prod after deploy, not sandbox (developer).
+- QA: check only, no test file added to the repo (developer, 2026-10-23).
+- No rebase / no merge of `_production` into the branch (no overlap; rebase would need force-push).
+- Wake-up behavior kept: a 30-day quiet issue that fires again goes back to normal escalation (can jump to HIGH).
+
+## Blockers
+Waiting on people: PR #208 approval on GitHub (auto-sets Back-end Review via `_Worker_Clickup_BeReview::FromPr`), Front-end Review field in ClickUp (should be N/A — no front-end code; reviewer unknown, developer to ask lead), then merge + prod deploy.
+
+## Exact next step
+> After worker2 is deployed to production: run `/start TRUE-82794`, then read-only prod checks — `Core.WorkerJobs` rows for action `Infrastructure/Errors/Escalate` (issuesChecked ~671, no `truncated` key, failures 0, `deferred` drops to 0 over ~7–10 runs), Logs.Issue id 36 (`2J`) urgency LOW, ClickUp task 868kkrybk priority Low, and the stuck-HIGH count (HIGH + minimumUrgency LOW + open task + unacknowledged + dtLastOccurred < NOW() - 30 DAY) goes from 327 to 0.
+
+---
+_Saved by /session-save on 2026-10-23_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1122",
+  "version": "1.0.1124",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",