npm - toga-ai - Versions diffs - 1.0.504 → 1.0.505 - Mend

toga-ai 1.0.504 → 1.0.505

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (17) hide show

package/knowledge/1.0/apps/library/INDEX.md CHANGED Viewed

@@ -8,6 +8,7 @@
 | [Diagnostic Dialog — View Recommended Services Routing](features/diagnostic-dialog-view-recommended-services.md) | Two "View Recommended Services" buttons exist in the TOGa Refresh 2026 SR view: 1. | library/app/model/toga/diagnostic.php, library/app/model/servicerequest.php |
 | [Elite Freshservice Sync (library)](features/elite-freshservice-sync.md) | `App_Api_Toga2` in `library/app/api/toga2.php` orchestrates bidirectional sync between TOGA 2 and TOGaDesk. | library/app/api/toga2.php |
 | [Branded HTML Email Templates (App_Email_Template)](features/email-templates.md) | `App_Email_Template` (`app/email/template.php`) is the base class for branded HTML emails in the 1.0 (`App_`) framework. | library/app/email/template.php, library/app/email/agilant.php |
+| [Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)](features/error-capture-1-0.md) | The 1.0 side of the platform error-reporting pipeline (TRUE-78188). | library/app/error/capture.php, library/app/error.php, library/app/exception/business.php, library/app/cloud.php, worker/config.worker.ini, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |
 | [1.0 MVC Page Pattern & New-App Skeleton](features/mvc-page-pattern-and-app-skeleton.md) | This is the **reusable recipe for standing up a new 1.0 (`App_`) application** and for adding pages to one — the folder-based MVC routing, the page lifecycle, t | library/app/framework.php, library/app/frameworkindex.php, library/app/mvc.php, library/app/database.php, library/app/model.php, library/app/config.php |
 | [isFulfillable from NetSuite during Item Sync (Phase 1)](features/netsuite-item-isfulfillable-sync.md) | This is the **1.0 (Phase 1)** half of the `isFulfillable` feature: reading the NetSuite `isfulfillable` flag during item sync and stamping it onto the **Agilant | library/app/netsuite.php, library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/backfill_isfulfillable_jul5.php |
 | [NetSuite SuiteQL/REST API Reference](features/netsuite-suiteql-api-reference.md) | General working reference for the Agilant NetSuite integration: how to authenticate, how SuiteQL behaves, and the confirmed schema of the tables/columns/codes w | library/app/api/netsuite/rest.php, library/ssl/netsuite_ec_key.pem, test/@dave/Junk Drawer/nsq.php |

package/knowledge/1.0/apps/library/features/error-capture-1-0.md ADDED Viewed

@@ -0,0 +1,165 @@
+---
+title: "Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)"
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-08-04
+owners: ["jcardinal"]
+files:
+  - library/app/error/capture.php
+  - library/app/error.php
+  - library/app/exception/business.php
+  - library/app/cloud.php
+  - worker/config.worker.ini
+  - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
+related:
+  - ../architecture.md
+  - ../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
+  - ../../../2.0/apps/worker2/features/error-escalation-cron.md
+  - ../../tools/features/errors-curation-console.md
+  - ../../standards/backend-php.md
+---
+## Summary
+The 1.0 side of the platform error-reporting pipeline (TRUE-78188). `App_Error_Capture`
+(`library/app/error/capture.php`) records 1.0 errors as occurrences in the **same shared 2.0 Logs
+database** (`Logs.Issue` / `IssueFingerprint` / `Event`) that 2.0 writes to, so 1.0 and 2.0 feed
+**one issue list, one Tools console, one escalation cron**. Before this, 1.0 errors existed only
+in Sentry and were invisible to the issue system.
+`App_Error_Capture::captureException()` is the single 1.0 capture entry point and returns the
+quotable `Issue.reference`, mirroring `_Error::captureException()` in 2.0. Add new 1.0 capture
+paths by calling it — never by re-implementing persistence.
+Read the 2.0 doc
+([error-reporting-issue-event](../../../2.0/apps/_underscore/features/error-reporting-issue-event.md))
+for the data model, fingerprint scheme, reference encoding, and the MySQL constraints. This doc
+covers only what is **different or additional in 1.0**.
+## Opt-in by config section
+Capture requires a **`[database_toga2logs]`** section in the app's config (added to
+`worker/config.worker.ini`). **An app with no such section silently does not capture.** That is
+deliberate: `library` is shared by every 1.0 app, and a shared library must not fatal or spam an
+app that has not been wired to the Logs DB yet. Wiring a new 1.0 app into error reporting is
+therefore purely a config change.
+> `dbname` is **not** necessarily `Logs` — 2.0 resolves the Core Logs database name at runtime
+> from `Core.Databases` id 11. Read the real name per environment (same caveat as the Tools
+> console).
+## How it works
+### Hand-written, fully escaped SQL
+1.0 has **no `_Model`**, so every statement in `capture.php` is hand-written against
+`App_Database`, with every interpolated value escaped (`App_Database::sqlEscape()`) or cast. There
+is no model layer to absorb a schema change here: a rename in the shared Logs DB must be applied
+to this file in the same release as the migration (the same exposure the Tools console has).
+### `App_Exception_Business` — one issue across both frameworks
+`library/app/exception/business.php` mirrors 2.0's `_Exception_Business`: a stable `issueKey` plus
+a `minimumUrgency` floor. **The fingerprint text is byte-identical to 2.0's** (`'issueKey: ' .
+KEY`), so the *same key* thrown from either framework resolves to **one** Issue with one
+occurrence count and one set of recipients. Do not "tidy" that string on either side — the
+cross-framework identity is the whole point.
+Business issues are titled by their key: `Logs.Issue.subject` holds the `issueKey` (e.g.
+`COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`) while `Issue.errorMessage` and each
+`Event.errorMessage` keep the per-occurrence text. Otherwise the issue ends up named after
+whichever occurrence happened to be seen first — i.e. one arbitrary order number.
+Compass USA's sales-order → MITS transmit rejection was the first business-exception use case.
+### Client attribution
+`App_Error::setCurrentClientId()` exists in 1.0 for parity with 2.0's ambient current-client, but
+**1.0 has no central hook** equivalent to `_Database::registerClientDatabases()`. Client-scoped
+1.0 crons therefore **set it explicitly** (see
+`worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php`). Capture
+prefers an explicit `clientId` argument and falls back to the ambient value.
+### Fingerprint fallback (no usable stack)
+Frames are keyed on **file + function**, not `file:line`, so inserting lines above a throw site
+does not split an issue. When there is no usable application stack (fatals, OOM), identity is
+**exception class + entry point (script basename) + message with digits masked** — never the
+exception's own file/line, which for an OOM is wherever the allocation happened to cross the
+limit. Same scheme as 2.0; keep the two implementations in lockstep.
+### The 1.0 "red box" error page
+The rendered error page now headlines the **issue reference** (e.g. `1A-1`). Sentry's event id is
+a fallback shown only when our own capture produced nothing. The `sleep(13)` that existed purely
+to let Sentry ingest before the link was clicked is gone. Business exceptions get a different
+headline and **do** render their message — safe for that class only, because a business message is
+written for a reader and carries no file, line, or trace.
+### `Logs.Event.context` shape
+Identical in both frameworks: **every top-level key holds an object**, and scalars are grouped
+under a `Reference` object. Top-level keys are the display names — `Reference`, `GET`, `POST`,
+`Runtime`, `AWS`, `SERVER`. `GET`/`POST` are cast to objects so an empty one serializes as `{}`,
+not `[]`. `$_POST` is captured and redacted on the same key-name basis as `$_GET` (note PHP only
+populates `$_POST` for form encodings, so JSON API bodies deliberately do not appear). The
+over-length shed list was renamed in lockstep and its marker is an object too.
+### Elastic Beanstalk environment name on AL1
+1.0 runs **"64bit Amazon Linux/2.8.8" (AL1)**, which predates
+`/opt/elasticbeanstalk/bin/get-config`, and the `042-write-eb-environment.sh` deploy hook only
+exists in `api2`/`worker2`. So 1.0 reads the AL1 container config at
+`/opt/elasticbeanstalk/deploy/configuration/containerconfiguration`, searching **recursively** for
+`environment_name` (its position moved between platform versions), then falls back to
+`aws ec2 describe-tags`.
+## Gotchas / known issues
+- **Any PHP warning inside capture code would kill the request.** 1.0's
+  `App_Error::handleError()` routes **every** PHP warning into `handleException()`, which renders
+  the red box and calls `exit()` — so a single warning raised while capturing (an unreachable IMDS
+  endpoint is the everyday case: `file_get_contents` warns before returning `false`) terminates
+  the request or cron mid-run while reporting an unrelated error. 2.0's equivalent handler
+  *throws*, which a `catch` absorbs. `captureException()` therefore sets
+  `App_Error::setThrowExceptionsEnabled(false)` for its own duration and restores it in
+  `finally`. This disables the **handler**, not real throws — `App_Query` raises a plain
+  `throw new Exception` on DB errors, so genuine failures still reach the catch. **This is a
+  general 1.0 trap for any code that must not escalate**, not just error capture.
+- **No typed properties anywhere in `library/`.** 1.0 runs **PHP 7.2**; a deploy failed with
+  `syntax error, unexpected '?'` on `private static ?string $memoryReserve`. Nine typed properties
+  across `error.php`, `error/capture.php`, and `exception/business.php` were converted to untyped
+  with `@var` docblocks. Parameter and return types are fine (7.0/7.1).
+- **An early `return` can make a fallback unreachable.** Two bugs in the first EB-environment
+  attempt: a `return` when `get-config` was not executable skipped the `describe-tags` fallback
+  entirely — i.e. the fallback was dead on the only platform that needed it — and the CLI call
+  omitted `App_Cloud::$preCommands`, so it did not authenticate the way every other 1.0 AWS call
+  does. **Unverified:** whether that IAM identity actually holds `ec2:DescribeTags`.
+- **Fingerprint scheme changes re-group existing issues once.** Moving to file+function frames and
+  the class+entryPoint fallback changes hashes, so open issues re-fingerprint on first occurrence
+  after deploy. Deliberate; merge old hashes onto the curated Issue with the Tools
+  merge-fingerprint box.
+- **Pre-existing committed secrets (location only — not fixed, out of scope).**
+  `library/app/cloud.php` holds a hardcoded **AWS access key id and secret access key** as
+  class constants/statics (~lines 23–24), used by `App_Cloud::$preCommands` for every 1.0 AWS CLI
+  call. Unrelated to this work. Treat as compromised: move to SSM like the other parameters and
+  rotate.
+## Change history
+- 2026-08-04 — Built the 1.0 side of TRUE-78188: `App_Error_Capture` writing occurrences into the
+  shared 2.0 `Logs` DB (opt-in via `[database_toga2logs]`, hand-written escaped SQL);
+  `App_Exception_Business` with a fingerprint string byte-identical to 2.0's so one `issueKey`
+  spans both frameworks; business issues titled by `issueKey`; the red-box page headlining the
+  issue reference and dropping the Sentry `sleep(13)`; the uniform object-per-top-level-key
+  `Event.context` shape with `$_POST` capture; AL1 container-config + `describe-tags` resolution of
+  the EB environment name; explicit `setCurrentClientId()` for client-scoped crons. Recorded two
+  framework-level traps: PHP 7.2 forbids typed properties in `library/`, and
+  `App_Error::handleError` escalates any warning to a rendered page + `exit()`, which capture
+  suppresses with `setThrowExceptionsEnabled(false)`/`finally`. Also fixed a latent `$this->errStr`
+  in a static branch of `error.php` and noted the pre-existing hardcoded AWS keys in
+  `app/cloud.php`. (jcardinal)

package/knowledge/1.0/apps/tools/INDEX.md CHANGED Viewed

@@ -7,7 +7,7 @@
 | [CloudFront Client Setup](features/cloudfront-client-setup.md) | An SSO-gated admin tool at **`/devops/cloudfront-clients`** in the Tools 1.0 app that onboards a client onto **CloudFront + Route 53 across multiple AWS account | tools/_/app/devops/cloudfront.php, tools/mvc/devops/cloudfront-clients/get.php, tools/mvc/devops/cloudfront-clients/post.php, tools/assets/js/cloudfront-clients.js, tools/assets/css/cloudfront-clients.css, tools/_/app/nav.php, tools/_/app/frameworkindex.php, tools/config.production.ini |
 | [Design Demo Admin](features/design-demo-admin.md) | A self-serve admin UI at **`/design`** in the SSO-protected **Tools** app that lets the design team publish self-contained "Claude Design" HTML exports as **ver | tools/_/app/design/github.php, tools/mvc/design/get.php, tools/mvc/design/post.php, tools/assets/css/design.css, tools/assets/js/design.js, tools/_/app/frameworkindex.php, tools/_/app/nav.php, tools/composer.json |
 | [Tools — Developers Folder (UUID & Password Generators)](features/developer-tools.md) | The first two tools shipped in the Tools app, both under the **Developers** folder and gated to personas **Development Team** / **TOGa Technology**. | tools/mvc/developers/uuid/get.php, tools/mvc/developers/password/get.php |
-| [/errors Curation Console (Tools → shared Core Logs DB)](features/errors-curation-console.md) | Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as a 1.0 Tools MVC page reading the **shared Core Logs DB** through | tools/mvc/errors/get.php, tools/mvc/errors/post.php, tools/_/app/nav.php, tools/config.production.ini |
+| [/errors Curation Console (Tools → shared Core Logs DB)](features/errors-curation-console.md) | Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as a 1.0 Tools MVC page reading the **shared Core Logs DB** through | tools/mvc/errors/get.php, tools/mvc/errors/post.php, tools/mvc/errors/issue/get.php, tools/assets/css/style.css, tools/_/app/nav.php, tools/config.production.ini |
 | [Legacy Email Notifier (tools /email-migration/notify)](features/legacy-email-notifier.md) | An SSO-gated admin tool at **`/email-migration/notify`** (nav group **Email Migration** > **Legacy Notifier**, personas `['TOGa Technology','Development Team']` | tools/mvc/email-migration/notify/get.php, tools/mvc/email-migration/notify/post.php, tools/_/app/nav.php |
 | [Tools MVC — Routing, CSRF & App_Database Access Patterns](features/mvc-data-access-patterns.md) | The load-bearing 1.0 (`App_`) framework conventions a developer needs when adding a page to the Tools app — URL routing, CSRF, and DB access through `App_Databa | tools/_/app/nav.php, tools/mvc/get.php, library/app/error.php |
 | [Tools Persona-Gated Navigation (App_Nav)](features/persona-gated-navigation.md) | `App_Nav` is the Tools app's two-level, **persona-gated** navigation. | tools/_/app/nav.php, tools/mvc/get.php |

package/knowledge/1.0/apps/tools/features/errors-curation-console.md CHANGED Viewed

@@ -6,11 +6,13 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-08-01
+updated: 2026-08-04
 owners: ["jcardinal"]
 files:
   - tools/mvc/errors/get.php
   - tools/mvc/errors/post.php
+  - tools/mvc/errors/issue/get.php
+  - tools/assets/css/style.css
   - tools/_/app/nav.php
   - tools/config.production.ini
 related:
@@ -18,6 +20,7 @@ related:
   - ./mvc-data-access-patterns.md
   - ../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
   - ../../../2.0/apps/worker2/features/error-escalation-cron.md
+  - ../../library/features/error-capture-1-0.md
 ---
 ## Summary
@@ -41,6 +44,35 @@ happening and were deliberately never ticketed. Nobody had that view before.
 - **Any save sets `isManaged`**, which does double duty: it protects the curated text from being
   overwritten by later occurrences, **and** it exempts the Issue from the cron's GC sweep.
+### Routing and pages
+Filesystem routing (1.0 Tools MVC): `mvc/errors/` = **`/errors`** (the listing) and
+`mvc/errors/issue/` = **`/errors/issue`** (the detail page).
+**Listing** — 25 per page with a filtered `COUNT` (it was previously a silent `LIMIT 200`, so
+older issues simply did not exist as far as a curator could tell); bold monospace Issue ID; an
+**EB Environment** column read from the newest event's context; a **Type** column showing
+`BUSINESS` or `TECHNICAL`; and search + three dropdown filters (including EB Environment) laid out
+as a **3-column grid** — the original wrapping flex row broke to one control per line.
+**`Type` needs no new column.** The presence of `Issue.issueKey` *is* the business/technical
+signal. Do not add a `type` column for this (see the 2.0 doc's "proposed and not built" note).
+**Issue detail** — a two-column split: technical evidence left, business curation (recipients,
+curated text, affected clients) right, with an always-present Type chip.
+- **Stack trace and context come from the MOST RECENT event**, not the Issue's first-seen
+  snapshot. The first occurrence is rarely the one you are debugging.
+- **Context renders as cards**, one per top-level context group, ordered largest-first by recursive
+  node count, each a collapsible JSON tree built on native `<details>`/`<summary>` — **no
+  JavaScript** — with typed value colouring and `[REDACTED]` called out. This only works because
+  capture now emits a uniform object-per-top-level-key shape.
+- A **view-layer** key humanizer (`elasticBeanstalkEnvironment` → "Elastic Beanstalk Environment";
+  ALL_CAPS `$_SERVER` keys left verbatim). It is presentation only — never rename keys at capture
+  time.
+- Hourly-occurrence chart; fingerprints table re-sized for the narrow column with an icon-only
+  Move button.
 **This page notifies nobody.** Escalation, ClickUp ticketing, and email are owned entirely by
 `_Worker_Infrastructure_Errors::Escalate`. Adding a notify action here would create a second,
 untracked alerting path.
@@ -80,11 +112,32 @@ They were renamed from plural on 2026-08-01, after the pipeline was already in p
 - **Cross-tenant surface.** The Core Logs DB is shared by every client, so this console can show
   one client's captured context to a viewer looking at another client's problem. It is
   internal-only and persona-gated for that reason — do not expose any part of it to a client.
+- **⚠ Never `JSON_EXTRACT` `Event.context`.** The column is `TEXT` and a context truncated at
+  `CONTEXT_MAX_LENGTH` is **invalid JSON** — `JSON_EXTRACT` would raise and take the whole page
+  down. The EB Environment filter therefore matches with **`SUBSTRING_INDEX`**, uses `EXISTS` (has
+  this issue *ever* occurred on that deployment), and validates the submitted value against the
+  discovered list — an allowlist, which also rules out `LIKE` wildcards.
+- **The distinct-EB-environment query is a full scan of `Event`.** Acceptable only because that
+  table is bounded by the 7-day purge. At volume the proper fix is denormalising the EB environment
+  onto its own `Event` column **at capture time**.
+- **Read both the old and new `Event.context` key positions.** The context shape was restructured
+  on 2026-08-04 (one object per top-level key, scalars under `Reference`); with 7-day retention,
+  both shapes coexist for a week after any such change.
 - **Curated text is authoritative.** Once `isManaged` is set, later occurrences no longer update
   subject/description, so a stale curated subject stays stale until someone edits it again.
 ## Change history
+- 2026-08-04 — Console build-out: the listing gained real pagination (25/page with a filtered
+  `COUNT`, replacing a silent `LIMIT 200`), an EB Environment column **and filter**, a
+  BUSINESS/TECHNICAL Type column derived from `issueKey` (no new DB column), and a 3-column filter
+  grid. New `/errors/issue` detail page split technical/curation into two columns, showing the
+  **most recent** event's trace and context, with context rendered as largest-first cards over a
+  JavaScript-free `<details>` JSON tree, a view-layer key humanizer, a Type chip, and an
+  hourly-occurrence chart. Recorded the load-bearing gotcha that `Event.context` must be matched
+  with `SUBSTRING_INDEX` and never `JSON_EXTRACT` (a truncated context is invalid JSON and would
+  take the page down), that the filter value is allowlisted against the discovered list, and that
+  the distinct-values scan is only safe because `Event` is bounded by the 7-day purge. (jcardinal)
 - 2026-08-01 — Updated every hardcoded table name in `tools/mvc/errors/get.php` (25+ refs) and
   `post.php` (15+ refs) for the Core Logs singular rename (`Issues`→`Issue`, `Events`→`Event`,
   `IssueFingerprints`→`IssueFingerprint`, `IssueClickupTasks`→`IssueClickupTask`,

package/knowledge/1.0/apps/tools/features/talos-kb-documents-admin.md CHANGED Viewed

@@ -6,7 +6,7 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-07-29
+updated: 2026-08-04
 owners: [jcardinal, ajean]
 files:
   - tools/mvc/talos/kb-documents/get.php
@@ -303,6 +303,13 @@ Refactor from a single hard-coded `development-team` KB to per-AI-model, data-dr
   idempotent, **fails the deploy loudly** if `pgsql` won't load) and
   `hooks/postdeploy/01-restart-php.sh` (restarts php-fpm/httpd). **EB runs `.platform/hooks`
   regardless of the git exec bit** — worker2's equivalents are committed `100644`.
+  > **Rewritten 2026-08-04 (in `tools` and `worker2`) — it failed on PHP 8.5.** The old hook tried
+  > the **generic** `php-pgsql` first, which resolves against the default 8.1 stream and can
+  > conflict; it also discarded `dnf` output and only checked that `pgsql` loaded, not
+  > **`pdo_pgsql`** (the extension `App_Pg` actually needs). It now tries the **versioned**
+  > candidate first, captures and prints `dnf` output so a failure is diagnosable from
+  > `eb-hooks.log`, checks **both** extensions, and honours a `PGSQL_HOOK_OPTIONAL=1` escape hatch
+  > for environments that must deploy without Postgres.
 - Apply the dbchanges2 `Team/2026-06-30a..e` + `Core/2026-06-30a` migrations in order.
 - Verify the Bedrock KB region / data-source name against the live account (TODOs in code).
@@ -397,6 +404,11 @@ Refactor from a single hard-coded `development-team` KB to per-AI-model, data-dr
 ## Change history
+- 2026-08-04 — Rewrote the `prebuild/01-install-php-pgsql.sh` hook (here and in `worker2`), which
+  failed on PHP 8.5: it tried the generic `php-pgsql` (resolving against the default 8.1 stream),
+  swallowed `dnf` output, and checked only `pgsql` rather than `pdo_pgsql`. Now tries the versioned
+  candidate first, prints `dnf` output, verifies both extensions, and supports
+  `PGSQL_HOOK_OPTIONAL=1`. (jcardinal)
 - 2026-07-29 — **Fixed the empty Vocabulary page: the per-model scoping shipped with no data
   backfill.** `vocabulary/get.php` reads strictly `WHERE c_trueAiModelId = <model>` (no NULL
   fallback), but `Team/2026-07-27a` added the column `DEFAULT NULL` and nothing populated it —

package/knowledge/1.0/apps/worker/INDEX.md CHANGED Viewed

@@ -8,6 +8,7 @@
 | [Elite TOGA 2.0 → TOGaDeskSupport Standalone Attachment Sync](features/elite-togadesk-attachment-sync.md) | `sync_togadesk_elite_attachments.php` is a standalone cron (every 5 minutes) that syncs file attachments from TOGA 2.0 into TOGaDeskSupport for Elite. | worker/crons/toga2/elite/sync_togadesk_elite_attachments.php, worker/crons/toga2/elite/test_sync_togadesk_elite_attachments.php |
 | [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/netsuite/rest.php, library/app/systemmonitor/netsuiteintegration.php |
+| [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2 platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went dow | worker/.ebextensions/040_disk_memory_hygiene.config, worker/.ebextensions/045_oneuptime_agent.config, worker/ebs/cron.worker.php, worker/ebs/mount-s3fs-folders.php, worker/ebs/apache_settings.php, worker/ebs/setup_phpini.php |
 | [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Diagnosing frozen 1.0 worker cron check-ins (Sentry "missed" flood)](workflows/diagnosing-frozen-cron-checkins.md) | When 1.0 worker cron timestamps freeze and Sentry project `worker1` fills with **`missed`** check-ins, the intuitive diagnosis — a wedged `App_Framework::isProc | worker/.ebextensions/cron.config, library/app/worker.php |

package/knowledge/1.0/apps/worker/features/oneuptime-server-monitor-host-hygiene.md ADDED Viewed

@@ -0,0 +1,204 @@
+---
+title: OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host
+framework: "1.0"
+repo: worker
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-08-03
+owners: ["jcardinal"]
+files:
+  - worker/.ebextensions/040_disk_memory_hygiene.config
+  - worker/.ebextensions/045_oneuptime_agent.config
+  - worker/ebs/cron.worker.php
+  - worker/ebs/mount-s3fs-folders.php
+  - worker/ebs/apache_settings.php
+  - worker/ebs/setup_phpini.php
+related: ["oneuptime-worker-uptime-monitoring"]
+---
+## Summary
+The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2
+platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went down from the EC2
+instance running **out of memory and out of disk**. Two things address it:
+1. **Host-level observability** — a OneUptime **Server monitor** per worker role, fed by the
+   OneUptime **Infrastructure Agent** running on the box (CPU / memory / disk / network /
+   process metrics every 30s). This is *not* a heartbeat — see
+   [[oneuptime-worker-uptime-monitoring]] for the separate Incoming-Request heartbeat layer.
+2. **Disk/memory hygiene** — a `.ebextensions` config that reaps stale files and rotates
+   logs (safe/additive), plus a set of **behavior-changing fixes that are documented but
+   NOT applied** (Apache prefork sizing, PHP `memory_limit`, s3fs `ensure_diskfree`,
+   cron `flock`).
+## Key files / entry points
+| File | Purpose | Risk |
+|---|---|---|
+| `.ebextensions/040_disk_memory_hygiene.config` | Deletes stale junk (s3fs `/tmp` cache, `/var/www/cache` staging, orphaned `WORKER_ERROR_*`, deploy litter) + rotates logs | Safe / additive — deletes only stale files |
+| `.ebextensions/045_oneuptime_agent.config` | Installs + supervises the OneUptime infrastructure-agent (static Go binary, v11.7.4) | Safe / additive — no app behavior change |
+## How the Server monitor works
+- The agent POSTs metrics every **30s** to
+  `{oneuptime_url}/server-monitor/response/ingest/{secret_key}`; OneUptime evaluates the
+  monitor's **Criteria** against them to decide Operational / Degraded / Offline.
+- **AL1 has no systemd**, so `045_…` deliberately skips the agent's `configure`/`start`
+  subcommands (they register a systemd/SysV service and fail here). It writes
+  `/etc/oneuptime-infrastructure-agent/config.json` directly and runs the foreground **`run`**
+  subcommand under a **1-minute cron watchdog** — the same systemd-bypass the ODP Voice
+  container monitor uses.
+- Metrics reported: CPU (aggregate + per-core + user/system/idle/iowait/steal breakdown),
+  load average 1/5/15, memory + swap, **disk per mount** (total/used/free/%, device, fstype,
+  I/O counters — criteria target a specific path), network (aggregate + per-interface bytes/
+  packets/errors/drops, TCP connection counts; cumulative since boot, OneUptime derives
+  throughput from deltas), host info (platform/kernel/uptime/virtualization), and
+  **per-process** pid/name/command/cpu%/mem%/status/threads/user (enables
+  "process is executing" criteria).
+## Provisioning — already done, per-role model
+`agilant-worker` is one EB Worker tier whose instances each claim ONE stable role
+(`ebs/cron.worker.php`), so there is **one Server monitor per role (7)**, grouped under the
+OneUptime label **`Worker 1.0`**. Each monitor already carries the criteria below, and its
+secret is **hardcoded in `045_oneuptime_agent.config`**, selected at runtime by the role in
+`/etc/worker-role`. Nothing to create — you only need to deploy.
+| Role | Monitor name |
+|------|--------------|
+| notification | Notification Worker 1.0 - System Health |
+| database | Database Worker 1.0 - System Health |
+| infrastructure | Infrastructure Worker 1.0 - System Health |
+| toga | TOGa Worker 1.0 - System Health |
+| togadesk | TOGa Desk Worker 1.0 - System Health |
+| sync | Sync Worker 1.0 - System Health |
+| catalog | Catalog Worker 1.0 - System Health |
+**Key rotation:** regenerate that monitor's secret in OneUptime and update the matching
+`case` arm in `045_…`. Instance-ids churn but roles are stable, so a recycled box
+re-reports to the same role monitor automatically.
+**Creating a monitor from scratch** (only if a new role is added): Monitors → Create Monitor
+→ type **Server / VM** → the secret key appears on the monitor's Settings tab after save.
+Via REST it is `POST /api/monitor` with `{"data":{"projectId":…,"name":…,"monitorType":"Server"}}`
+and an `ApiKey` header — but the secret is server-generated and **not** returned in the
+create body; read it from Settings.
+**Verify after deploy**, on the instance:
+```bash
+pgrep -fa oneuptime-infrastructure-agent      # process running
+tail -n 40 /var/log/oneuptime-agent.log       # agent log
+```
+Last Ping updates within ~1 min.
+## Criteria (order matters — first match wins, catch-all Operational last)
+| Order | Result state | Check | Condition | Threshold |
+|---|---|---|---|---|
+| 1 | Offline | CPU Usage % | Greater Than | 90 |
+| 2 | Offline | Memory Usage % | Greater Than | 90 |
+| 3 | Degraded | Disk Usage % (path `/`) | Greater Than | 90 |
+| 4 | Degraded | CPU Usage % | Greater Than | 75 |
+| 5 | Degraded | Memory Usage % | Greater Than | 75 |
+| 6 | Operational (fallback) | CPU Usage % | Less Than Or Equal To | 100 |
+Each criterion's filter condition is **All**. Recommended additions for this worker
+specifically: Degraded if disk `/` > **80%** (earlier warning — disk is this box's actual
+failure mode), Degraded if swap % used > 50% (early OOM signal once swap exists), and
+Offline if the Apache process is not executing (Server Process → Command contains `httpd`
+→ Is Not Executing).
+Via API, criteria live in `monitorSteps.monitorCriteria.criteriaInstances[]`; each instance
+has `filters[]` (`checkOn`, operator, `value`, plus `serverMonitorOptions.diskPath` for disk
+checks), a `filterCondition` (`All`/`Any`), and a `monitorStatusId`.
+## Root cause of the outages
+### Disk
+| Cause | Evidence | Status |
+|---|---|---|
+| s3fs `use_cache=/tmp` **unbounded**, never evicted | `ebs/mount-s3fs-folders.php:53`, `ebs/s3fs.json` (3 buckets) | **Not fixed** — needs `ensure_diskfree` (below) |
+| `/var/www/cache` 0777 with no reaper; DB dumps up to **12 GB/part**, deleted only on successful upload | `ebs/setup_export_cache_folders.php:11-13`, `crons/database/database_backup.php:75,314,337` | Reaper cron shipped in `040_…` |
+| `WORKER_ERROR_*` reaper never scheduled | `crons/worker/check_worker_errors.php` — present in **no** `schedules/*.json` | Reaper cron shipped in `040_…` |
+| Deploy litter (`/tmp/phpredis*`, s3fs source) | `010_setup_redis.config:3-6`, `006_mount-s3fs.config:16-29` | One-off cleanup shipped in `040_…` |
+### Memory
+| Cause | Evidence | Status |
+|---|---|---|
+| Apache prefork massively over-provisioned: `MaxRequestWorkers = ceil(3200*3/37) = 260` while PHP `memory_limit = 2G` | `ebs/apache_settings.php:3,8` + `ebs/setup_phpini.php:22` | **Not fixed** |
+| Long CLI cron jobs overlap (minute-ly schedules, no `flock`), each entitled to 2 GB | `ebs/cron.worker.php`, `schedules/cron.worker*.json` (26–103 jobs) | **Not fixed** |
+| Standing band-aid: Apache `graceful` every 20 min "to close db connections" | `schedules/cron.worker.json:3-7` | **Not fixed** — would be retired by `MaxConnectionsPerChild` |
+## Fixes documented but NOT applied — review before applying
+These change app behavior. Apply deliberately.
+**Bound the s3fs cache** — `ebs/mount-s3fs-folders.php`, after the `use_cache` option is
+appended (line 53):
+```php
+$mount_command .= ' -o ' . escapeshellarg('ensure_diskfree=2000'); // MB; s3fs evicts its own cache to keep >=2GB free
+```
+**Right-size Apache** — `ebs/apache_settings.php`: measure the real child size (the command
+is in the file at line 4 — with `memory_limit=2G`, heavy endpoints are far bigger than the
+assumed 37 MB), then replace the `× 3` overcommit on line 8 with
+`MaxRequestWorkers = floor(availableRAM_MB * 0.7 / measuredChildMB)` (≈20–25 on a 4 GB box,
+not 260) and lower `StartServers`/`MinSpareServers` to 4–8. Add `MaxConnectionsPerChild 500`
+inside the `<IfModule prefork.c>` block so children recycle and return leaked memory to the
+OS — this is what retires the 20-minute graceful-restart hack.
+**Lower cron memory pressure** — `ebs/setup_phpini.php`: drop the global
+`memory_limit = 2G` (line 22) to `512M` and raise it per-script via `ini_set()` only in the
+few CLI batch jobs that need it; set `display_errors = off` (line 20) in production, logging
+to a rotated file instead.
+**Stop cron overlap** — where `ebs/cron.worker.php` builds each cron command, prefix
+minute-ly/2-minute jobs with a non-blocking lock:
+`flock -n /tmp/lock.<jobname> php /var/www/html/crons/<jobname>`.
+**Swap (optional stopgap)** — only *after* the disk reapers are in place; this box is
+disk-constrained and a 2 GB swapfile on a near-full root volume makes disk worse:
+```yaml
+commands:
+  02_swap:
+    test: "[ ! -f /var/swapfile ]"
+    command: "dd if=/dev/zero of=/var/swapfile bs=1M count=2048 && chmod 600 /var/swapfile && mkswap /var/swapfile && swapon /var/swapfile"
+```
+## Gotchas
+- **Disk path `/` works here** — this is a real EC2 host, unlike the ODP Voice container
+  case where "Disk metrics for path / were unavailable" appears because the container root
+  isn't in the reported disk list. s3fs (fuse) mounts may **not** be reported as normal
+  partitions, so prefer alerting on `/` rather than the s3fs mount.
+- The agent is kept alive by a **cron watchdog, not a service** — if it stops reporting,
+  check the watchdog and `/var/log/oneuptime-agent.log`, not `service`/`systemctl`.
+- Monitor secrets are hardcoded per-role in `045_…`; treat that file as a credential
+  location. Do not copy secrets into docs, logs, or other files.
+- **Security follow-up (urgent, out of scope of the monitoring work):** hardcoded live AWS
+  credentials and DB passwords are committed in plaintext in `ebs/cron.worker.php:14-15`,
+  `crons/worker/worker_heartbeat.php:12`, `crons/database/database_backup*.php`, and
+  `config.worker.ini`. Rotate and move to EB environment properties / an IAM instance role.
+## References
+- Agent + subcommands — https://github.com/OneUptime/oneuptime/blob/master/InfrastructureAgent/README.md
+- Server monitor — https://oneuptime.com/docs/monitor/server-monitor
+- Criteria instance API — https://oneuptime.com/reference/monitor-criteria-instance
+- Monitor REST API — https://oneuptime.com/reference/monitor
+- Agent releases (v11.7.4) — https://github.com/OneUptime/oneuptime/releases
+## Change history
+- 2026-08-03 — Captured from `worker/docs/oneuptime-server-monitor.md` (that in-repo doc is
+  being deleted; this is now the source of truth). Documents the per-role OneUptime Server
+  monitors + infrastructure agent (`045_…`), the shipped disk/memory hygiene reapers
+  (`040_…`), the diagnosed disk/memory root causes, and the reviewed-but-unapplied
+  behavior-changing fixes. (jcardinal)

package/knowledge/1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md CHANGED Viewed

@@ -11,7 +11,7 @@ owners: ["jcardinal"]
 files:
   - library/app/worker.php
   - worker/crons/worker/worker_heartbeat.php
-related: []
+related: ["oneuptime-server-monitor-host-hygiene"]
 ---
 ## Summary

package/knowledge/1.0/standards/backend-php.md CHANGED Viewed

@@ -5,11 +5,12 @@ project: Library
 client: shared
 type: standard
 status: active
-updated: 2026-07-24
+updated: 2026-08-04
 owners: [jcardinal, rgirish, mhammontree]
 files: []
 related:
   - ../apps/library/architecture.md
+  - ../apps/library/features/error-capture-1-0.md
   - ../../2.0/standards/backend-php.md
 ---
@@ -259,6 +260,49 @@ if ($condition) {
 $piece = substr($value, 0, 8);
 ```
+### No typed properties (PHP 7.2)
+* 1.0 runs **PHP 7.2**, so **typed properties (7.4+) are forbidden anywhere in `library/` or a 1.0
+  app.** Declare class attributes untyped with an `@var` docblock:
+```php
+// WRONG in 1.0 — deploy fails with "syntax error, unexpected '?'"
+private static ?string $memoryReserve = null;
+// CORRECT in 1.0
+/** @var string|null */
+private static $memoryReserve = null;
+```
+* **Parameter and return types ARE fine** (PHP 7.0/7.1) — this rule is only about *properties*.
+* This is the opposite of the 2.0 standard, which requires typed properties. The universal
+  `rules/toga/common/coding-style.md` rule is already **scoped to 2.0** and cross-references this
+  standard for the 1.0 `@var` docblock form — so there is no conflict to resolve, just two different
+  rules for two different runtimes.
+* Real failure: a deploy of `library/app/error/capture.php` broke on
+  `private static ?string $memoryReserve`; nine typed properties across three files had to be
+  converted.
+### Any PHP warning terminates the request — how to opt out locally
+`App_Error::handleError()` routes **every** PHP warning into `handleException()`, which renders the
+error page and calls **`exit()`**. (2.0's equivalent *throws*, which a `catch` can absorb — 1.0 does
+not give you that.) So a warning raised inside code that must not escalate — error capture, a
+monitoring ping, an IMDS probe where `file_get_contents` warns before returning `false`, a retry
+loop — **kills the request or cron mid-run while reporting an unrelated error.**
+Two sanctioned patterns, in order of preference:
+1. **A local handler around the risky call** — `set_error_handler()` throwing an `ErrorException`,
+   then `restore_error_handler()`. Use this when you want to *catch and retry* (the NetSuite
+   SSL-drop retry loop does this).
+2. **`App_Error::setThrowExceptionsEnabled(false)` with a `finally` restore** — use this when the
+   code must never escalate at all (`App_Error_Capture::captureException()` does this for its whole
+   duration). This disables the **handler**, not real `throw`s: `App_Query` raises a plain
+   `throw new Exception` on a DB error, so genuine failures still reach your `catch`.
+Never leave either one in effect beyond the block that needs it.
 ### Documentation
 * **DocBlocks:** framework (`library`) classes/methods are well documented with a description and `@param`/`@return`/`@throws`. Application code is less consistent — document new and modified methods to the framework standard.

package/knowledge/2.0/apps/_underscore/INDEX.md CHANGED Viewed

@@ -17,7 +17,7 @@
 | [Re-pointing a DB alias mid-request (_Database::register park/restore)](features/database-alias-repointing.md) | `_Database` keys **all live per-database runtime state by the connection ALIAS** (`Client` / `_underscore::DB_CLIENT`, `ClientLogs`, `Archive`), **not** by the | _underscore/Database.php, _underscore/Query.php, api2/Component/Api/V2/V2.php, api2/Component/Api/CrossClient/CrossClient.php |
 | [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. | _underscore/Email.php, worker2/Worker/Infrastructure/Email/Send.php |
 | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
-| [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Exception/Business.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Logs/2026-08-01a - Rename tables to singular.sql, dbchanges2/Logs_Client/2026-08-01a - Drop Error table.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
+| [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Database.php, _underscore/Exception/Business.php, api2/Controller/Index.php, worker2/Controller/Index.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Logs/2026-08-01a - Rename tables to singular.sql, dbchanges2/Logs_Client/2026-08-01a - Drop Error table.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
 | [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
 | [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | worker2/Component/Forecast/SaleImport/SaleImport.php, worker2/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/fixer.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
 | [isFulfillable Propagation Up the SO↔PO Chain](features/fulfillable-item-propagation.md) | `Items.isFulfillable` is a boolean that gates whether a storefront line's **Qty Fulfilled** cell is actionable. | _underscore/Model/Client/Item.php, _underscore/Model/Compass/Item.php, dbchanges2/Core/2026-07-17 - Items isFulfillable RecordField.sql, dbchanges2/Core/2026-07-17 - RegisterItemIsFulfillableInterceptors.sql, dbchanges2/Client/2026-07-17 - ItemsisFulfillable.sql |

package/knowledge/2.0/apps/_underscore/features/error-reporting-issue-event.md CHANGED Viewed

@@ -6,11 +6,14 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-03
+updated: 2026-08-04
 owners: ["dfranks", "jcardinal"]
 files:
   - _underscore/Error.php
+  - _underscore/Database.php
   - _underscore/Exception/Business.php
+  - api2/Controller/Index.php
+  - worker2/Controller/Index.php
   - _underscore/Model/Core/Logs/Issue.php
   - _underscore/Model/Core/Logs/Event.php
   - _underscore/Model/Core/Logs/IssueFingerprint.php
@@ -27,6 +30,8 @@ related:
   - ./email-send-pipeline.md
   - ../../worker2/features/error-escalation-cron.md
   - ../../../1.0/apps/tools/features/errors-curation-console.md
+  - ../../../1.0/apps/library/features/error-capture-1-0.md
+  - ../../api2/features/v2-api-error-codes.md
   - ../../dbchanges2/architecture.md
 ---
@@ -76,9 +81,25 @@ Issue itself. Anything that reads like "this client's issue" is a modelling erro
 ### Fingerprinting (identity)
-The fingerprint is the normalized top-N **application** frames (repo-relative `path:line`) plus
-the exception class; framework/vendor plumbing is filtered out. The exception **message is
-deliberately excluded** — it carries per-occurrence data (ids, values) and would defeat dedup.
+The fingerprint is the normalized top-N **application** frames plus the exception class;
+framework/vendor plumbing is filtered out. The exception **message is deliberately excluded** — it
+carries per-occurrence data (ids, values) and would defeat dedup.
+**A frame is FILE + FUNCTION, never `file:line`** (changed 2026-08-04). Line numbers made the
+identity hostage to unrelated edits: a 10-line insertion above a throw site in
+`Component/Api/V2/V2.php` split one failure into two Issues (`1E` and `1I`).
+**Fallback when there is no usable application stack** (fatals, OOM): **exception class + entry
+point (script basename) + the message with digits masked.** It must *not* use the exception's own
+file/line — for an OOM that is wherever the allocation happened to cross the limit, so issues `1F`
+and `1G` were the same memory exhaustion one line apart in `_underscore/Model.php`, a file that
+**is** in the skip list but which the old fallback never consulted.
+> Both changes alter existing hashes, so open Issues **re-fingerprint once** on their next
+> occurrence after deploy. That one-time re-grouping is deliberate; merge the old hashes onto the
+> curated Issue with the Tools merge-fingerprint box. The 1.0 implementation
+> ([error-capture-1-0](../../../1.0/apps/library/features/error-capture-1-0.md)) uses the same
+> scheme — keep the two in lockstep.
 This replaced `md5(serialize(debug_backtrace()))`, which serialized **call arguments** and so
 produced a different hash on every request — dedup never actually worked.
@@ -92,7 +113,7 @@ from the Tools console to merge them.
 | Thrown class | HTTP | Sentry | `Logs.Issue` | Fingerprint / routing |
 |---|---|---|---|---|
 | `_Exception_Validation` | **400** | no | **no** | n/a — bad client input. Still leaves a full `Logs.Api` row (see [api2 request logging](../../api2/features/request-logging.md)) |
-| `_Exception_Business` | **500** | yes | **yes** | stable `issueKey` (refactor-immune) → business recipients configured in the Tools console |
+| `_Exception_Business` | **4xx** (was 500 before 2026-08-04) | yes | **yes** | stable `issueKey` (refactor-immune) → business recipients configured in the Tools console. **Now a subclass of `_Exception_Validation` — must be caught FIRST, see below** |
 | plain global `Exception` | **500** | yes | **yes** | **trace**-based fingerprint (moves when code moves) → developer ClickUp task |
 | `_Exception` | — | — | — | **never throw it deliberately** (below) |
@@ -113,12 +134,62 @@ exception is ever constructed, so the real signal is destroyed. Jeff states this
 `_underscore/Exception/Business.php` carries a stable `issueKey` plus a `minimumUrgency` floor.
 When an `issueKey` is declared, **the key is the fingerprint**, so the Issue's identity does not
-depend on line numbers at all. It extends `Exception` (not `_Exception`), matching
-`_Exception_Validation`.
+depend on line numbers at all. It extends **`_Exception_Validation`** as of 2026-08-04 (never
+`_Exception`) — see the ordering trap below.
 It **deliberately never throws on a bad argument** — it normalizes and `error_log()`s instead.
 Replacing the caller's business error with an argument error destroys the real signal.
+**A business Issue is titled by its key.** `Issue.subject` holds the `issueKey` (e.g.
+`COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`); `Issue.errorMessage` and each `Event.errorMessage`
+keep the per-occurrence text. Otherwise the Issue is named after whichever occurrence was seen
+first — i.e. one arbitrary order number.
+`library`'s `App_Exception_Business` is the 1.0 twin and builds a **byte-identical** fingerprint
+string (`'issueKey: ' . KEY`), so one key thrown from either framework is **one** Issue with one
+occurrence count and one recipient set. Do not reformat that string on either side.
+#### ⚠ `_Exception_Business` now extends `_Exception_Validation` (decided 2026-08-04)
+It previously extended `Exception`. Extending `_Exception_Validation` makes api2 answer **4xx**
+instead of 500 for a business condition while **still recording it**.
+**The ordering trap:** `instanceof _Exception_Validation` is now **true** for a business
+exception. Any handler must test **business first**, or business exceptions silently stop being
+captured. This applies to api2's front-controller catch chain *and* to the three existing
+`catch (_Exception_Validation $e)` blocks in `_Model_Prudential_ServiceRequest` — and to any new
+`catch (_Exception_Validation …)` anyone writes. api2's `INTERNAL_SERVER_ERROR` message is also
+now guarded behind `!$isClientError`, so a business exception no longer stacks a 500 on its 400.
+### Ambient "current client" for attribution
+`_Error::setCurrentClientId()` / `getCurrentClientId()` hold an ambient client id, set
+automatically inside **`_Database::registerClientDatabases()`** — the one place in 2.0 that always
+knows the client, called with an already-resolved id from every client-scoped path (all
+JWT-authenticated V2 sites, cXML, per-client workers). `captureException()` prefers an explicit
+`clientId` argument and falls back to the ambient one. **No JWT decoding is needed:** by the time
+`registerClientDatabases()` runs, the JWT's client uuid has already been resolved to an id.
+This fixed Issue `1I`: a Compass PUT whose log went to `Logs_Compass` recorded `clientId` NULL,
+because api2's catch only had `$api->client->id`, which is populated on the login/refresh paths
+but **not** on an ordinary authenticated request. Never re-derive the client in a catch block —
+read the ambient value.
+### `Event.context` shape (uniform, both frameworks)
+**Every top-level key holds an object**; scalars are grouped under a `Reference` object. The
+top-level keys *are* the display names: `Reference`, `GET`, `POST`, `Runtime`, `AWS`, `SERVER`.
+`GET`/`POST` are cast to objects so an empty one serializes as `{}` rather than `[]`. `$_POST` is
+now captured (redacted on the same key-name basis as `$_GET`; PHP only populates `$_POST` for form
+encodings, so JSON API bodies deliberately do not appear). The duplicate top-level
+`peakMemoryMb` was dropped, the over-length shed list was renamed in lockstep with its marker now
+an object, and 2.0 gained `framework` / `isBusiness` / `issueKey` / `errorFile` / `errorLine` for
+1.0 parity.
+A uniform shape is what lets a viewer render context **generically** (see the Tools console's
+card/JSON-tree renderer). Consumers must read **both** the old and new key positions during the
+7-day `Event` retention overlap.
 ### The quotable reference
 `Issues.reference` is a Sentry-style base-26 encoding of `Issues.id`: 1 → `1A`, 26 → `1Z`,
@@ -172,9 +243,11 @@ error reporting with no overlap means going blind. Three findings make removal i
    warnings, but that promotion must be verified **empirically** — if it does not survive,
    code that currently 500s would silently continue with a null value, which is worse.
-Also: this pipeline is **2.0-only**. Sentry must remain for the 1.0 apps (desk1, retail1,
-forecast1, worker1, view1, commerce1) until a `library` / `App_Error` port lands, so the
-Sentry-removal ticket cannot complete until the 1.0 port does.
+**The 1.0 port has landed (2026-08-04)** — `App_Error_Capture` writes into this same shared `Logs`
+DB, so the pipeline is no longer 2.0-only; see
+[error-capture-1-0](../../../1.0/apps/library/features/error-capture-1-0.md). Each 1.0 app still
+has to opt in by adding a `[database_toga2logs]` config section, so Sentry must remain for any 1.0
+app that has not been wired up yet.
 ## Gotchas / known issues
@@ -221,6 +294,23 @@ Sentry-removal ticket cannot complete until the 1.0 port does.
   `varchar(255)` (`errorMessage`, `subject`) **throws** on `->save()`. Inside the handler's
   `catch(Throwable)` the throw is swallowed and the row is silently dropped — precisely when a
   long message matters most. Truncate to column width before `save()`.
+- **Hand-written SQL is what a table rename actually breaks — and it fails *after* the model
+  succeeds.** The singular rename left plural literals in hand-written statements across four
+  files, so `$issue->save()` succeeded and then `UPDATE Issues …` threw **1146** before the
+  fingerprint insert — creating a **new Issue per occurrence**: 70 duplicate `Logs.Issue` rows in
+  production. After any rename in a shared DB, grep for the literal name in *every* repo
+  (`_underscore`, `worker2`, `tools`), not just the models.
+- **Capture must force reads to the writer for its whole duration.** `_Model::save()` re-reads the
+  row, which inside an open transaction throws *"Exactly 1 row was expected to be returned but 0
+  were"* when the read goes to the read host. Capture sets
+  `_Database::setIsReadHostEnabled(false)` for the entire block (the same read-your-writes fix
+  `_Email::send()` uses, TRUE-80448).
+- **`Issue.reference` cannot be a MySQL generated column** (error 3109 — a generated column cannot
+  refer to an auto-increment column), which is exactly why the column is nullable and written by a
+  follow-up `UPDATE`. Do not "improve" it into a generated column.
+- **Unrelated, open (found via this pipeline, 2026-08-04):** `Logs.Api` inserts are failing in
+  production with *"Column 'isAuthRequest' cannot be null"*, surfaced as issue `2G` from
+  `worker-production`. Not fixed — needs its own ticket.
 - **`const TABLE` must track a table rename, and nothing warns you.** A `_Model` subclass whose
   `const TABLE` points at a renamed table fails only at query time. When a `Logs` table is
   renamed, the matching `_underscore/Model/Core/Logs/*.php` `const TABLE` must change in the
@@ -289,6 +379,22 @@ clientUserId). **Neither was built.** As built instead:
 ## Change history
+- 2026-08-04 — **Decided: `_Exception_Business` extends `_Exception_Validation`** so api2 answers
+  4xx for a business condition while still recording it — with the ordering trap that every
+  `catch (_Exception_Validation …)` must now test business **first** (api2's controller and the
+  three blocks in `_Model_Prudential_ServiceRequest`), plus api2's `INTERNAL_SERVER_ERROR` message
+  guarded behind `!$isClientError`. **Fingerprinting reworked:** frames are FILE + FUNCTION (a
+  10-line insertion in `V2.php` had split one failure into issues `1E`/`1I`) and the no-stack
+  fallback is class + entryPoint + digit-masked message (issues `1F`/`1G` were one OOM in
+  `_underscore/Model.php` one line apart) — a deliberate one-time re-fingerprinting of open issues.
+  Added the **ambient current client** (`_Error::setCurrentClientId()`, set inside
+  `_Database::registerClientDatabases()`), fixing issue `1I`'s NULL `clientId` on ordinary
+  authenticated api2 requests. Restructured `Event.context` to one object per top-level key with
+  scalars under `Reference` (identical in 1.0), added `$_POST` capture, and titled business issues
+  by `issueKey`. Recorded the 70-duplicate-Issue production incident (plural table literals in
+  hand-written SQL throwing 1146 after `save()` succeeded), the writer-read requirement, the 3109
+  reason `reference` is PHP-written, and the unrelated open `Logs.Api isAuthRequest` failure.
+  The 1.0 port (`App_Error_Capture`) landed, so this pipeline is no longer 2.0-only. (jcardinal)
 - 2026-08-03 — Added the **exception-class routing table** (`_Exception_Validation` → 400/no
   Issue; `_Exception_Business` → 500 + `issueKey`-fingerprinted Issue to business recipients;
   plain `Exception` → 500 + trace-fingerprinted Issue to a developer task) and recorded that

package/knowledge/2.0/apps/api2/features/v2-api-error-codes.md CHANGED Viewed

@@ -152,6 +152,24 @@ guard and swaps the real backend message for a generic fallback:
 Verification gap: `toga2-supply` had no `node_modules`, so `tsc` could not be run.
 **`npm run build` is the definitive check** — the type change will surface anything missed.
+## ⚠ Catch-order trap: a business exception IS a validation exception (2026-08-04)
+`_Exception_Business` now **extends `_Exception_Validation`** (it previously extended `Exception`),
+so a business condition answers **4xx** instead of 500 while still being recorded in the Issue
+pipeline. The consequence:
+> **`$e instanceof _Exception_Validation` is now TRUE for a business exception.** Every catch chain
+> must test **business first**. Get the order wrong and business exceptions silently stop being
+> captured — no error, no ticket, no email.
+This applies to `api2/Controller/Index.php` and to any `catch (_Exception_Validation $e)` block
+anywhere (the three in `_Model_Prudential_ServiceRequest` were audited for it). Also: the
+controller's `INTERNAL_SERVER_ERROR` message is gated behind `!$isClientError`, so a business
+exception no longer stacks a 500 message onto its 400 response.
+Full mechanics:
+[error-reporting-issue-event](../../_underscore/features/error-reporting-issue-event.md).
 ## Related recipes
 - **Add a writable field to a V2 record (3-file migration)** and **expose a scripted API**
@@ -164,6 +182,12 @@ Verification gap: `toga2-supply` had no `node_modules`, so `tsc` could not be ru
 ## Change history
+- 2026-08-04 — Recorded the catch-order trap from `_Exception_Business` now extending
+  `_Exception_Validation` (business conditions answer 4xx and are still captured, but any
+  `catch (_Exception_Validation …)` must test business **first** or they go unrecorded), and the
+  `INTERNAL_SERVER_ERROR` message now gated behind `!$isClientError`. Also noted that api2 no
+  longer needs to derive the client in its catch block — capture reads the ambient client id set by
+  `_Database::registerClientDatabases()`. (jcardinal)
 - 2026-07-30 — **BREAKING:** envelope `error` changed from a bare string to
   `{ code, id }`, where `id` is the quotable Core Logs reference for a captured exception and
   `null` for a graceful rejection (always present, for a predictable shape); assembled from

package/knowledge/2.0/apps/worker2/INDEX.md CHANGED Viewed

@@ -3,7 +3,7 @@
 | Doc | Summary | Files |
 |-----|---------|-------|
 | [Worker (worker2) Architecture](architecture.md) | Worker (repo `worker2`) is an AWS Elastic Beanstalk **Worker Tier** application that processes background jobs. | worker2/Controller/Index.php, worker2/Worker/, worker2/LambdaFunctions/, _underscore/Worker.php, worker2/composer.json |
-| [Deploy-Time Auto-Registration to the Shared ALB Target Group (non-production)](features/alb-target-group-auto-registration.md) | TOGA does **not** pay for EB-managed load-balancer registration, so an EB instance is normally **not** added to its environment's ALB target group — a fresh or | worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, worker2/ebs/register_instance_to_shared_application_load_balancer.php, worker2/.platform/hooks/prebuild/_shared/040-write-instance-id.sh, worker2/.platform/hooks/prebuild/_shared/041-write-region.sh, worker2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
+| [Deploy-Time Auto-Registration to the Shared ALB Target Group (non-production)](features/alb-target-group-auto-registration.md) | TOGA does **not** pay for EB-managed load-balancer registration, so an EB instance is normally **not** added to its environment's ALB target group — a fresh or | worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, worker2/ebs/register_instance_to_shared_application_load_balancer.php, worker2/.platform/hooks/prebuild/_shared/040-write-instance-id.sh, worker2/.platform/hooks/prebuild/_shared/041-write-region.sh, worker2/.platform/hooks/prebuild/_shared/042-write-eb-environment.sh, api2/.platform/hooks/prebuild/_shared/042-write-eb-environment.sh, worker2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
 | [All-Client Email Queue Monitor (Monitor/Operations/EmailQueue)](features/all-client-email-queue-monitor.md) | `_Worker_Monitor_Operations::EmailQueue()` is the cross-client health check for the 2.0 outbound email queue (`Logs_<Client>.Email` — see [2.0 Email Send Pipeli | worker2/Worker/Monitor/Operations.php, _underscore/Database.php, _underscore/Query.php, dbchanges2/Logs_Client/2026-05-21 - Email.sql |
 | [Automated PR Merger — Concurrent Force-Push Clobber Race](features/automated-pr-merger-force-push-race.md) | The automated PR merger `_Worker_Team_GitHub::Merge` (`worker2` `Worker/Team/Github.php`) merges approved PRs to `_production` by **force-pushing from a clone t | Worker/Team/Github.php |
 | [Callback Scheduling ("call me back") — worker2 AI-BDR](features/callback-scheduling.md) | The **AI-BDR callback path**: what happens between a prospect saying *"call me back later"* on a Vapi call and the dialer actually placing that second call. | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php, worker2/Controller/Index.php |
@@ -18,7 +18,7 @@
 | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
-| [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Config/production.ini, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
+| [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Controller/Index.php, worker2/Config/production.ini, _underscore/Model/Core/Logs/Issue.php, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql, dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql |
 | [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |
 | [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
 | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |

package/knowledge/2.0/apps/worker2/features/alb-target-group-auto-registration.md CHANGED Viewed

@@ -6,13 +6,15 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-28
+updated: 2026-08-04
 owners: [jcardinal]
 files:
   - worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
   - worker2/ebs/register_instance_to_shared_application_load_balancer.php
   - worker2/.platform/hooks/prebuild/_shared/040-write-instance-id.sh
   - worker2/.platform/hooks/prebuild/_shared/041-write-region.sh
+  - worker2/.platform/hooks/prebuild/_shared/042-write-eb-environment.sh
+  - api2/.platform/hooks/prebuild/_shared/042-write-eb-environment.sh
   - worker2/.platform/hooks/postdeploy/015_install_composer.sh
   - api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
   - api2/ebs/register_instance_to_shared_application_load_balancer.php
@@ -64,10 +66,27 @@ invocation**, not through SQS.
 |---|---|
 | `prebuild/_shared/040-write-instance-id.sh` | `storage/instance-id.txt` |
 | `prebuild/_shared/041-write-region.sh` | `storage/region.txt` |
+| `prebuild/_shared/042-write-eb-environment.sh` | `storage/eb-environment.txt` |
 | `postdeploy/015_install_composer.sh` | `vendor/` (the AWS SDK) |
 Renumber it below `015` and the SDK autoloader does not exist yet.
+### `042-write-eb-environment.sh` — the EB environment name, cached to disk (2026-08-04)
+Added to the same `_shared` hook family (plus a postdeploy hook) in **both `api2` and `worker2`**,
+writing the EB environment name to disk. It resolves it from `get-config`, then
+`$EB_ENVIRONMENT_NAME`, then the IMDS tag.
+**Why cache it instead of shelling out when needed:** the first consumer is
+[error capture](../../_underscore/features/error-reporting-issue-event.md), which also runs from the
+**shutdown handler during a fatal — possibly an OOM**. Forking a process at that moment is the
+least likely thing to succeed. Same reasoning as `040`/`041`: resolve at deploy time, read a file
+at runtime.
+1.0 has no equivalent hook (its AL1 platform has no `get-config`), so `library` resolves the name
+at runtime instead — see
+[error-capture-1-0](../../../1.0/apps/library/features/error-capture-1-0.md).
 ### Porting prerequisites (all already satisfied in worker2)
 The port needed **no adaptation** because `worker2` already had: `aws/aws-sdk-php` in
@@ -153,6 +172,11 @@ which lists the other hooks but not `060` — pending an elevated-doc update.
 ## Change history
+- 2026-08-04 — Added `_shared/042-write-eb-environment.sh` (+ a postdeploy hook) to **api2 and
+  worker2**, caching the EB environment name to disk from `get-config` → `$EB_ENVIRONMENT_NAME` →
+  the IMDS tag. Read from a file rather than shelled out to at use time because its first consumer
+  is error capture, which also runs from the shutdown handler during a fatal/OOM where a fork is
+  least likely to succeed. (jcardinal)
 - 2026-07-28 — Ported the hook pair from `api2` into `worker2` (new files `060_….sh` +
   `ebs/register_instance_to_shared_application_load_balancer.php`) so non-production worker2
   environments are reachable through the shared ALB for manual HTTP invocation. Hardened vs. the

package/knowledge/2.0/apps/worker2/features/error-escalation-cron.md CHANGED Viewed

@@ -6,17 +6,21 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-01
+updated: 2026-08-04
 owners: ["jcardinal"]
 files:
   - worker2/Worker/Infrastructure/Errors.php
   - worker2/Worker/Clickup/ErrorTask.php
   - worker2/Worker/Clickup.php
+  - worker2/Controller/Index.php
   - worker2/Config/production.ini
+  - _underscore/Model/Core/Logs/Issue.php
   - dbchanges2/Core/2026-07-30a - Error escalation cron job.sql
+  - dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql
 related:
   - ../../_underscore/features/error-reporting-issue-event.md
   - ../../../1.0/apps/tools/features/errors-curation-console.md
+  - ../../../1.0/apps/library/features/error-capture-1-0.md
   - ./creating-worker-actions.md
   - ./clickup-project-routing.md
 ---
@@ -34,9 +38,12 @@ through this one action, so there is exactly one place to mute or debug alerting
 ### Urgency = max of two independent axes, plus a floor
-- **Volume** — occurrences in a 10-minute window: **20 / 50 / 100**.
+- **Volume** — occurrences in a **1-hour** window: **NORMAL 5 / HIGH 10 / URGENT 20** (retuned
+  2026-08-04 from 20/50/100 over 10 minutes — a 10-minute window never fired for the
+  low-and-steady failures that actually matter).
 - **Neglect** — hours an *unacknowledged* ClickUp task has sat: **1 / 4 / 24**.
-- **Floor** — the Issue's declared `minimumUrgency` applies under *both* axes.
+- **Sustained-24h** — an issue still occurring a day later escalates on duration alone.
+- **Floor** — the Issue's declared `minimumUrgency` applies under *all* axes.
 The neglect axis exists because volume alone cannot see a rare-but-serious error, nor a serious
 error that everyone is ignoring.
@@ -44,9 +51,21 @@ error that everyone is ignoring.
 ### Fast up, slow down
 Escalation applies **the same run** a band is crossed. De-escalation requires the count to sit
-under a **lower release band (12 / 35 / 70)** for **3 consecutive windows**. Without that gap,
+under a **lower release band** for **`REQUIRED_QUIET_RUNS`** consecutive runs. Without that gap,
 an issue hovering near a threshold flaps every minute and burns a ClickUp write each time.
+> The constant was renamed `REQUIRED_QUIET_WINDOWS` → **`REQUIRED_QUIET_RUNS`** because it counts
+> **cron runs**, not windows. The DB column `Issue.consecutiveQuietWindows` still carries the old
+> name — that mismatch is known and intentional (renaming a live column was not worth it).
+### What was actually pushed to ClickUp is stored, not inferred
+`Issue.clickupPriority` (migration `dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql`)
+records the priority the cron **last pushed**. The old logic used "did our computed urgency
+change?" as a proxy and rewrote the priority every minute — logging *"changed priority from high to
+high"*. Storing what was pushed also makes it **self-healing** when someone edits the priority in
+ClickUp by hand. Replaying a production run went from **8 PUTs to 2**.
 ### Task-creation gate
 A ClickUp task is created only when urgency ≥ NORMAL, **or** a floor is set, **or** the Issue is
@@ -83,6 +102,20 @@ ClickUp is the acknowledge/resolve system — developers already live there, and
 Tools would just be ignored. **Any** status change on an error task counts as acknowledgement,
 which is safe only because the automation never sets status itself.
+> **60-second creation grace window (fix, 2026-08-04).** ClickUp fires a status webhook for a
+> task's *initial* status, so the automation was acknowledging its **own** newly created tasks —
+> stamping `dtAcknowledged` ~1s after `dtCreated` and permanently disabling the neglect axis for
+> that issue. Status webhooks arriving within 60s of creation are ignored.
+### Notification and task presentation
+- The technical escalation email to **devteam@togatech.com** now **coexists** with the ClickUp
+  update. It was an `elseif`, so only one of the two ever fired.
+- ClickUp titles carry the **reference only** — `[1E]`, not `[1E-1]`. Including the occurrence
+  number made the title churn on every escalation.
+- **Occurrences** and **Last Occurrence** custom fields are synced, and task bodies open with a
+  **"Debug in Tools"** deep link.
 ### Area-level owner learning (observation only)
 `IssueAreaOwner` learns an owner after **5 in a row** and ships **observation-only** — a hint in
@@ -115,8 +148,39 @@ recipients away** after a quiet month.
 - **`[clickup] sprint_folder_id`** — added to `worker2/Config/production.ini`, left **empty with
   instructions**. Note `[clickup]` exists in only **2 of the 8** worker2 configs.
+## Worker job failures are captured in the front controller, not by the handler
+**worker2's front controller catches `Throwable` itself** (exactly like api2), writes the message
+into `Core.WorkerJobs.output`, and returns 200 — so `_Error::exceptionHandler` **never fires** and
+job failures went entirely uncaptured. Deploying `_underscore` changed nothing, because nothing
+called capture.
+`worker2/Controller/Index.php` now calls `_Error::captureException()` at **six** catch sites
+(three per duplicated branch: DB-logs registration, `initialize()`, and the action itself). Each
+call:
+- happens **after** `_Database::transactionRollback()` — a capture write inside a doomed
+  transaction is lost;
+- happens **before** the `WorkerJobs` UPDATE, so the reference is **prepended** to `output`. A
+  failed job row reads `[1Q-25] …`, letting you jump from `Core.WorkerJobs` straight to the Issue
+  in Tools;
+- is **individually guarded** — capture must never be able to convert a handled job failure into
+  an unhandled one.
+Diagnosed from `Core.WorkerJobs` id 583346 (`Team/Transcripts/Export`). **Any new catch site in
+this controller needs the same treatment.**
 ## Gotchas / known issues
+- **Persist state before, or independently of, any outbound call.** A regression chain took the
+  whole run down every minute: `syncCustomFields()` read `$issue->dtLastOccurred`, which
+  `loadWorkingSet()` never selected; `_Error::errorHandler()` promotes the undefined-property
+  warning to an **exception**, which killed the run **right after the ClickUp PUT and before
+  `persistIssueState()`** — so urgency never persisted and every run recomputed the same "change".
+  Three fixes: select the missing columns; wrap `processTechnicalIssue()` in try/catch so state
+  **always** persists even when an outbound call fails; and store `Issue.clickupPriority` instead
+  of inferring what was pushed. **A missing column in a working-set SELECT is a runtime exception
+  in 2.0, not a null.**
 - **The working-set query must be a UNION, not an OR.** It is a UNION of three separately
   indexable branches. MySQL will **not** `index_merge` across an `OR` when one branch is a
   correlated `EXISTS`, so the OR form **full-scans `Issue` every 60 seconds, forever**.
@@ -143,6 +207,24 @@ recipients away** after a quiet month.
 ## Change history
+- 2026-08-04 — **Fixed: worker job failures were never captured at all** — worker2's front
+  controller catches `Throwable` and returns 200, so six catch sites in
+  `worker2/Controller/Index.php` now call `_Error::captureException()` after rollback and before the
+  `WorkerJobs` UPDATE, prepending the reference to `output`. **Fixed the every-minute priority
+  rewrite** ("changed priority from high to high"): `loadWorkingSet()` was missing
+  `dtLastOccurred`, and the promoted undefined-property warning killed the run after the ClickUp
+  PUT but before `persistIssueState()`; added the columns, wrapped `processTechnicalIssue()` so
+  state always persists, and replaced the urgency-change proxy with a stored
+  `Issue.clickupPriority` (new migration `dbchanges2/Logs/2026-08-03a`) — 8 PUTs down to 2 on a
+  replayed run. **Fixed the automation acknowledging its own new tasks** (ClickUp emits a status
+  webhook for the initial status, which froze the neglect axis) with a 60s creation grace window.
+  Retuned escalation: 1-hour window, thresholds 5/10/20 with release bands, a new sustained-24h
+  axis, `REQUIRED_QUIET_WINDOWS` → `REQUIRED_QUIET_RUNS` (the DB column keeps the old name),
+  ClickUp titles dropped the occurrence number (`[1E-1]` → `[1E]`), Occurrences/Last Occurrence
+  custom fields synced, a "Debug in Tools" deep link in task bodies, and the devteam escalation
+  email now coexists with the ClickUp update (was an `elseif`, so only one ever fired). Also fixed
+  plural table literals left in hand-written SQL here after the singular rename — part of the
+  70-duplicate-Issue incident. (jcardinal)
 - 2026-08-01 — Table names updated for the Core Logs **singular** rename (`Issue`, `Event`,
   `IssueFingerprint`, `IssueClickupTask`, `IssueEmailAddress`, `IssueAreaOwner`). Naming-only;
   the cron's behavior is unchanged. See

package/knowledge/INDEX.md CHANGED Viewed

@@ -4,8 +4,8 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
-- **library** (Library) _(framework core)_ — 14 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 18 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **library** (Library) _(framework core)_ — 15 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
+- **worker** (Worker) — 19 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 11 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)

package/package.json CHANGED Viewed

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.504",
+  "version": "1.0.505",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",