toga-ai 1.0.504 → 1.0.506

package/knowledge/1.0/apps/library/INDEX.md

@@ -8,6 +8,7 @@
 | [Diagnostic Dialog — View Recommended Services Routing](features/diagnostic-dialog-view-recommended-services.md) | Two "View Recommended Services" buttons exist in the TOGa Refresh 2026 SR view: 1. | library/app/model/toga/diagnostic.php, library/app/model/servicerequest.php |
 | [Elite Freshservice Sync (library)](features/elite-freshservice-sync.md) | `App_Api_Toga2` in `library/app/api/toga2.php` orchestrates bidirectional sync between TOGA 2 and TOGaDesk. | library/app/api/toga2.php |
 | [Branded HTML Email Templates (App_Email_Template)](features/email-templates.md) | `App_Email_Template` (`app/email/template.php`) is the base class for branded HTML emails in the 1.0 (`App_`) framework. | library/app/email/template.php, library/app/email/agilant.php |
+| [Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)](features/error-capture-1-0.md) | The 1.0 side of the platform error-reporting pipeline (TRUE-78188). | library/app/error/capture.php, library/app/error.php, library/app/exception/business.php, library/app/cloud.php, worker/config.worker.ini, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |
 | [1.0 MVC Page Pattern & New-App Skeleton](features/mvc-page-pattern-and-app-skeleton.md) | This is the **reusable recipe for standing up a new 1.0 (`App_`) application** and for adding pages to one — the folder-based MVC routing, the page lifecycle, t | library/app/framework.php, library/app/frameworkindex.php, library/app/mvc.php, library/app/database.php, library/app/model.php, library/app/config.php |
 | [isFulfillable from NetSuite during Item Sync (Phase 1)](features/netsuite-item-isfulfillable-sync.md) | This is the **1.0 (Phase 1)** half of the `isFulfillable` feature: reading the NetSuite `isfulfillable` flag during item sync and stamping it onto the **Agilant | library/app/netsuite.php, library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/backfill_isfulfillable_jul5.php |
 | [NetSuite SuiteQL/REST API Reference](features/netsuite-suiteql-api-reference.md) | General working reference for the Agilant NetSuite integration: how to authenticate, how SuiteQL behaves, and the confirmed schema of the tables/columns/codes w | library/app/api/netsuite/rest.php, library/ssl/netsuite_ec_key.pem, test/@dave/Junk Drawer/nsq.php |

package/knowledge/1.0/apps/library/features/error-capture-1-0.md

@@ -0,0 +1,165 @@
+---
+title: "Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)"
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-08-04
+owners: ["jcardinal"]
+files:
+  - library/app/error/capture.php
+  - library/app/error.php
+  - library/app/exception/business.php
+  - library/app/cloud.php
+  - worker/config.worker.ini
+  - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
+related:
+  - ../architecture.md
+  - ../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
+  - ../../../2.0/apps/worker2/features/error-escalation-cron.md
+  - ../../tools/features/errors-curation-console.md
+  - ../../standards/backend-php.md
+---
+
+## Summary
+
+The 1.0 side of the platform error-reporting pipeline (TRUE-78188). `App_Error_Capture`
+(`library/app/error/capture.php`) records 1.0 errors as occurrences in the **same shared 2.0 Logs
+database** (`Logs.Issue` / `IssueFingerprint` / `Event`) that 2.0 writes to, so 1.0 and 2.0 feed
+**one issue list, one Tools console, one escalation cron**. Before this, 1.0 errors existed only
+in Sentry and were invisible to the issue system.
+
+`App_Error_Capture::captureException()` is the single 1.0 capture entry point and returns the
+quotable `Issue.reference`, mirroring `_Error::captureException()` in 2.0. Add new 1.0 capture
+paths by calling it — never by re-implementing persistence.
+
+Read the 2.0 doc
+([error-reporting-issue-event](../../../2.0/apps/_underscore/features/error-reporting-issue-event.md))
+for the data model, fingerprint scheme, reference encoding, and the MySQL constraints. This doc
+covers only what is **different or additional in 1.0**.
+
+## Opt-in by config section
+
+Capture requires a **`[database_toga2logs]`** section in the app's config (added to
+`worker/config.worker.ini`). **An app with no such section silently does not capture.** That is
+deliberate: `library` is shared by every 1.0 app, and a shared library must not fatal or spam an
+app that has not been wired to the Logs DB yet. Wiring a new 1.0 app into error reporting is
+therefore purely a config change.
+
+> `dbname` is **not** necessarily `Logs` — 2.0 resolves the Core Logs database name at runtime
+> from `Core.Databases` id 11. Read the real name per environment (same caveat as the Tools
+> console).
+
+## How it works
+
+### Hand-written, fully escaped SQL
+
+1.0 has **no `_Model`**, so every statement in `capture.php` is hand-written against
+`App_Database`, with every interpolated value escaped (`App_Database::sqlEscape()`) or cast. There
+is no model layer to absorb a schema change here: a rename in the shared Logs DB must be applied
+to this file in the same release as the migration (the same exposure the Tools console has).
+
+### `App_Exception_Business` — one issue across both frameworks
+
+`library/app/exception/business.php` mirrors 2.0's `_Exception_Business`: a stable `issueKey` plus
+a `minimumUrgency` floor. **The fingerprint text is byte-identical to 2.0's** (`'issueKey: ' .
+KEY`), so the *same key* thrown from either framework resolves to **one** Issue with one
+occurrence count and one set of recipients. Do not "tidy" that string on either side — the
+cross-framework identity is the whole point.
+
+Business issues are titled by their key: `Logs.Issue.subject` holds the `issueKey` (e.g.
+`COMPASS_SALES_ORDER_MITS_TRANSMIT_REJECTED`) while `Issue.errorMessage` and each
+`Event.errorMessage` keep the per-occurrence text. Otherwise the issue ends up named after
+whichever occurrence happened to be seen first — i.e. one arbitrary order number.
+
+Compass USA's sales-order → MITS transmit rejection was the first business-exception use case.
+
+### Client attribution
+
+`App_Error::setCurrentClientId()` exists in 1.0 for parity with 2.0's ambient current-client, but
+**1.0 has no central hook** equivalent to `_Database::registerClientDatabases()`. Client-scoped
+1.0 crons therefore **set it explicitly** (see
+`worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php`). Capture
+prefers an explicit `clientId` argument and falls back to the ambient value.
+
+### Fingerprint fallback (no usable stack)
+
+Frames are keyed on **file + function**, not `file:line`, so inserting lines above a throw site
+does not split an issue. When there is no usable application stack (fatals, OOM), identity is
+**exception class + entry point (script basename) + message with digits masked** — never the
+exception's own file/line, which for an OOM is wherever the allocation happened to cross the
+limit. Same scheme as 2.0; keep the two implementations in lockstep.
+
+### The 1.0 "red box" error page
+
+The rendered error page now headlines the **issue reference** (e.g. `1A-1`). Sentry's event id is
+a fallback shown only when our own capture produced nothing. The `sleep(13)` that existed purely
+to let Sentry ingest before the link was clicked is gone. Business exceptions get a different
+headline and **do** render their message — safe for that class only, because a business message is
+written for a reader and carries no file, line, or trace.
+
+### `Logs.Event.context` shape
+
+Identical in both frameworks: **every top-level key holds an object**, and scalars are grouped
+under a `Reference` object. Top-level keys are the display names — `Reference`, `GET`, `POST`,
+`Runtime`, `AWS`, `SERVER`. `GET`/`POST` are cast to objects so an empty one serializes as `{}`,
+not `[]`. `$_POST` is captured and redacted on the same key-name basis as `$_GET` (note PHP only
+populates `$_POST` for form encodings, so JSON API bodies deliberately do not appear). The
+over-length shed list was renamed in lockstep and its marker is an object too.
+
+### Elastic Beanstalk environment name on AL1
+
+1.0 runs **"64bit Amazon Linux/2.8.8" (AL1)**, which predates
+`/opt/elasticbeanstalk/bin/get-config`, and the `042-write-eb-environment.sh` deploy hook only
+exists in `api2`/`worker2`. So 1.0 reads the AL1 container config at
+`/opt/elasticbeanstalk/deploy/configuration/containerconfiguration`, searching **recursively** for
+`environment_name` (its position moved between platform versions), then falls back to
+`aws ec2 describe-tags`.
+
+## Gotchas / known issues
+
+- **Any PHP warning inside capture code would kill the request.** 1.0's
+  `App_Error::handleError()` routes **every** PHP warning into `handleException()`, which renders
+  the red box and calls `exit()` — so a single warning raised while capturing (an unreachable IMDS
+  endpoint is the everyday case: `file_get_contents` warns before returning `false`) terminates
+  the request or cron mid-run while reporting an unrelated error. 2.0's equivalent handler
+  *throws*, which a `catch` absorbs. `captureException()` therefore sets
+  `App_Error::setThrowExceptionsEnabled(false)` for its own duration and restores it in
+  `finally`. This disables the **handler**, not real throws — `App_Query` raises a plain
+  `throw new Exception` on DB errors, so genuine failures still reach the catch. **This is a
+  general 1.0 trap for any code that must not escalate**, not just error capture.
+- **No typed properties anywhere in `library/`.** 1.0 runs **PHP 7.2**; a deploy failed with
+  `syntax error, unexpected '?'` on `private static ?string $memoryReserve`. Nine typed properties
+  across `error.php`, `error/capture.php`, and `exception/business.php` were converted to untyped
+  with `@var` docblocks. Parameter and return types are fine (7.0/7.1).
+- **An early `return` can make a fallback unreachable.** Two bugs in the first EB-environment
+  attempt: a `return` when `get-config` was not executable skipped the `describe-tags` fallback
+  entirely — i.e. the fallback was dead on the only platform that needed it — and the CLI call
+  omitted `App_Cloud::$preCommands`, so it did not authenticate the way every other 1.0 AWS call
+  does. **Unverified:** whether that IAM identity actually holds `ec2:DescribeTags`.
+- **Fingerprint scheme changes re-group existing issues once.** Moving to file+function frames and
+  the class+entryPoint fallback changes hashes, so open issues re-fingerprint on first occurrence
+  after deploy. Deliberate; merge old hashes onto the curated Issue with the Tools
+  merge-fingerprint box.
+- **Pre-existing committed secrets (location only — not fixed, out of scope).**
+  `library/app/cloud.php` holds a hardcoded **AWS access key id and secret access key** as
+  class constants/statics (~lines 23–24), used by `App_Cloud::$preCommands` for every 1.0 AWS CLI
+  call. Unrelated to this work. Treat as compromised: move to SSM like the other parameters and
+  rotate.
+
+## Change history
+
+- 2026-08-04 — Built the 1.0 side of TRUE-78188: `App_Error_Capture` writing occurrences into the
+  shared 2.0 `Logs` DB (opt-in via `[database_toga2logs]`, hand-written escaped SQL);
+  `App_Exception_Business` with a fingerprint string byte-identical to 2.0's so one `issueKey`
+  spans both frameworks; business issues titled by `issueKey`; the red-box page headlining the
+  issue reference and dropping the Sentry `sleep(13)`; the uniform object-per-top-level-key
+  `Event.context` shape with `$_POST` capture; AL1 container-config + `describe-tags` resolution of
+  the EB environment name; explicit `setCurrentClientId()` for client-scoped crons. Recorded two
+  framework-level traps: PHP 7.2 forbids typed properties in `library/`, and
+  `App_Error::handleError` escalates any warning to a rendered page + `exit()`, which capture
+  suppresses with `setThrowExceptionsEnabled(false)`/`finally`. Also fixed a latent `$this->errStr`
+  in a static branch of `error.php` and noted the pre-existing hardcoded AWS keys in
+  `app/cloud.php`. (jcardinal)

package/knowledge/1.0/apps/tools/INDEX.md

@@ -7,7 +7,7 @@
 | [CloudFront Client Setup](features/cloudfront-client-setup.md) | An SSO-gated admin tool at **`/devops/cloudfront-clients`** in the Tools 1.0 app that onboards a client onto **CloudFront + Route 53 across multiple AWS account | tools/_/app/devops/cloudfront.php, tools/mvc/devops/cloudfront-clients/get.php, tools/mvc/devops/cloudfront-clients/post.php, tools/assets/js/cloudfront-clients.js, tools/assets/css/cloudfront-clients.css, tools/_/app/nav.php, tools/_/app/frameworkindex.php, tools/config.production.ini |
 | [Design Demo Admin](features/design-demo-admin.md) | A self-serve admin UI at **`/design`** in the SSO-protected **Tools** app that lets the design team publish self-contained "Claude Design" HTML exports as **ver | tools/_/app/design/github.php, tools/mvc/design/get.php, tools/mvc/design/post.php, tools/assets/css/design.css, tools/assets/js/design.js, tools/_/app/frameworkindex.php, tools/_/app/nav.php, tools/composer.json |
 | [Tools — Developers Folder (UUID & Password Generators)](features/developer-tools.md) | The first two tools shipped in the Tools app, both under the **Developers** folder and gated to personas **Development Team** / **TOGa Technology**. | tools/mvc/developers/uuid/get.php, tools/mvc/developers/password/get.php |
-| [/errors Curation Console (Tools → shared Core Logs DB)](features/errors-curation-console.md) | Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as a 1.0 Tools MVC page reading the **shared Core Logs DB** through | tools/mvc/errors/get.php, tools/mvc/errors/post.php, tools/_/app/nav.php, tools/config.production.ini |
+| [/errors Curation Console (Tools → shared Core Logs DB)](features/errors-curation-console.md) | Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as a 1.0 Tools MVC page reading the **shared Core Logs DB** through | tools/mvc/errors/get.php, tools/mvc/errors/post.php, tools/mvc/errors/issue/get.php, tools/assets/css/style.css, tools/_/app/nav.php, tools/config.production.ini |
 | [Legacy Email Notifier (tools /email-migration/notify)](features/legacy-email-notifier.md) | An SSO-gated admin tool at **`/email-migration/notify`** (nav group **Email Migration** > **Legacy Notifier**, personas `['TOGa Technology','Development Team']` | tools/mvc/email-migration/notify/get.php, tools/mvc/email-migration/notify/post.php, tools/_/app/nav.php |
 | [Tools MVC — Routing, CSRF & App_Database Access Patterns](features/mvc-data-access-patterns.md) | The load-bearing 1.0 (`App_`) framework conventions a developer needs when adding a page to the Tools app — URL routing, CSRF, and DB access through `App_Databa | tools/_/app/nav.php, tools/mvc/get.php, library/app/error.php |
 | [Tools Persona-Gated Navigation (App_Nav)](features/persona-gated-navigation.md) | `App_Nav` is the Tools app's two-level, **persona-gated** navigation. | tools/_/app/nav.php, tools/mvc/get.php |

package/knowledge/1.0/apps/tools/features/errors-curation-console.md

@@ -6,11 +6,13 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-08-01
+updated: 2026-08-04
 owners: ["jcardinal"]
 files:
   - tools/mvc/errors/get.php
   - tools/mvc/errors/post.php
+  - tools/mvc/errors/issue/get.php
+  - tools/assets/css/style.css
   - tools/_/app/nav.php
   - tools/config.production.ini
 related:
@@ -18,6 +20,7 @@ related:
   - ./mvc-data-access-patterns.md
   - ../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
   - ../../../2.0/apps/worker2/features/error-escalation-cron.md
+  - ../../library/features/error-capture-1-0.md
 ---
 
 ## Summary
@@ -41,6 +44,35 @@ happening and were deliberately never ticketed. Nobody had that view before.
 - **Any save sets `isManaged`**, which does double duty: it protects the curated text from being
   overwritten by later occurrences, **and** it exempts the Issue from the cron's GC sweep.
 
+### Routing and pages
+
+Filesystem routing (1.0 Tools MVC): `mvc/errors/` = **`/errors`** (the listing) and
+`mvc/errors/issue/` = **`/errors/issue`** (the detail page).
+
+**Listing** — 25 per page with a filtered `COUNT` (it was previously a silent `LIMIT 200`, so
+older issues simply did not exist as far as a curator could tell); bold monospace Issue ID; an
+**EB Environment** column read from the newest event's context; a **Type** column showing
+`BUSINESS` or `TECHNICAL`; and search + three dropdown filters (including EB Environment) laid out
+as a **3-column grid** — the original wrapping flex row broke to one control per line.
+
+**`Type` needs no new column.** The presence of `Issue.issueKey` *is* the business/technical
+signal. Do not add a `type` column for this (see the 2.0 doc's "proposed and not built" note).
+
+**Issue detail** — a two-column split: technical evidence left, business curation (recipients,
+curated text, affected clients) right, with an always-present Type chip.
+
+- **Stack trace and context come from the MOST RECENT event**, not the Issue's first-seen
+  snapshot. The first occurrence is rarely the one you are debugging.
+- **Context renders as cards**, one per top-level context group, ordered largest-first by recursive
+  node count, each a collapsible JSON tree built on native `<details>`/`<summary>` — **no
+  JavaScript** — with typed value colouring and `[REDACTED]` called out. This only works because
+  capture now emits a uniform object-per-top-level-key shape.
+- A **view-layer** key humanizer (`elasticBeanstalkEnvironment` → "Elastic Beanstalk Environment";
+  ALL_CAPS `$_SERVER` keys left verbatim). It is presentation only — never rename keys at capture
+  time.
+- Hourly-occurrence chart; fingerprints table re-sized for the narrow column with an icon-only
+  Move button.
+
 **This page notifies nobody.** Escalation, ClickUp ticketing, and email are owned entirely by
 `_Worker_Infrastructure_Errors::Escalate`. Adding a notify action here would create a second,
 untracked alerting path.
@@ -80,11 +112,32 @@ They were renamed from plural on 2026-08-01, after the pipeline was already in p
 - **Cross-tenant surface.** The Core Logs DB is shared by every client, so this console can show
   one client's captured context to a viewer looking at another client's problem. It is
   internal-only and persona-gated for that reason — do not expose any part of it to a client.
+- **⚠ Never `JSON_EXTRACT` `Event.context`.** The column is `TEXT` and a context truncated at
+  `CONTEXT_MAX_LENGTH` is **invalid JSON** — `JSON_EXTRACT` would raise and take the whole page
+  down. The EB Environment filter therefore matches with **`SUBSTRING_INDEX`**, uses `EXISTS` (has
+  this issue *ever* occurred on that deployment), and validates the submitted value against the
+  discovered list — an allowlist, which also rules out `LIKE` wildcards.
+- **The distinct-EB-environment query is a full scan of `Event`.** Acceptable only because that
+  table is bounded by the 7-day purge. At volume the proper fix is denormalising the EB environment
+  onto its own `Event` column **at capture time**.
+- **Read both the old and new `Event.context` key positions.** The context shape was restructured
+  on 2026-08-04 (one object per top-level key, scalars under `Reference`); with 7-day retention,
+  both shapes coexist for a week after any such change.
 - **Curated text is authoritative.** Once `isManaged` is set, later occurrences no longer update
   subject/description, so a stale curated subject stays stale until someone edits it again.
 
 ## Change history
 
+- 2026-08-04 — Console build-out: the listing gained real pagination (25/page with a filtered
+  `COUNT`, replacing a silent `LIMIT 200`), an EB Environment column **and filter**, a
+  BUSINESS/TECHNICAL Type column derived from `issueKey` (no new DB column), and a 3-column filter
+  grid. New `/errors/issue` detail page split technical/curation into two columns, showing the
+  **most recent** event's trace and context, with context rendered as largest-first cards over a
+  JavaScript-free `<details>` JSON tree, a view-layer key humanizer, a Type chip, and an
+  hourly-occurrence chart. Recorded the load-bearing gotcha that `Event.context` must be matched
+  with `SUBSTRING_INDEX` and never `JSON_EXTRACT` (a truncated context is invalid JSON and would
+  take the page down), that the filter value is allowlisted against the discovered list, and that
+  the distinct-values scan is only safe because `Event` is bounded by the 7-day purge. (jcardinal)
 - 2026-08-01 — Updated every hardcoded table name in `tools/mvc/errors/get.php` (25+ refs) and
   `post.php` (15+ refs) for the Core Logs singular rename (`Issues`→`Issue`, `Events`→`Event`,
   `IssueFingerprints`→`IssueFingerprint`, `IssueClickupTasks`→`IssueClickupTask`,

package/knowledge/1.0/apps/tools/features/talos-kb-documents-admin.md

@@ -6,7 +6,7 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-07-29
+updated: 2026-08-04
 owners: [jcardinal, ajean]
 files:
   - tools/mvc/talos/kb-documents/get.php
@@ -303,6 +303,13 @@ Refactor from a single hard-coded `development-team` KB to per-AI-model, data-dr
   idempotent, **fails the deploy loudly** if `pgsql` won't load) and
   `hooks/postdeploy/01-restart-php.sh` (restarts php-fpm/httpd). **EB runs `.platform/hooks`
   regardless of the git exec bit** — worker2's equivalents are committed `100644`.
+  > **Rewritten 2026-08-04 (in `tools` and `worker2`) — it failed on PHP 8.5.** The old hook tried
+  > the **generic** `php-pgsql` first, which resolves against the default 8.1 stream and can
+  > conflict; it also discarded `dnf` output and only checked that `pgsql` loaded, not
+  > **`pdo_pgsql`** (the extension `App_Pg` actually needs). It now tries the **versioned**
+  > candidate first, captures and prints `dnf` output so a failure is diagnosable from
+  > `eb-hooks.log`, checks **both** extensions, and honours a `PGSQL_HOOK_OPTIONAL=1` escape hatch
+  > for environments that must deploy without Postgres.
 - Apply the dbchanges2 `Team/2026-06-30a..e` + `Core/2026-06-30a` migrations in order.
 - Verify the Bedrock KB region / data-source name against the live account (TODOs in code).
 
@@ -397,6 +404,11 @@ Refactor from a single hard-coded `development-team` KB to per-AI-model, data-dr
 
 ## Change history
 
+- 2026-08-04 — Rewrote the `prebuild/01-install-php-pgsql.sh` hook (here and in `worker2`), which
+  failed on PHP 8.5: it tried the generic `php-pgsql` (resolving against the default 8.1 stream),
+  swallowed `dnf` output, and checked only `pgsql` rather than `pdo_pgsql`. Now tries the versioned
+  candidate first, prints `dnf` output, verifies both extensions, and supports
+  `PGSQL_HOOK_OPTIONAL=1`. (jcardinal)
 - 2026-07-29 — **Fixed the empty Vocabulary page: the per-model scoping shipped with no data
   backfill.** `vocabulary/get.php` reads strictly `WHERE c_trueAiModelId = <model>` (no NULL
   fallback), but `Team/2026-07-27a` added the column `DEFAULT NULL` and nothing populated it —

package/knowledge/1.0/apps/worker/INDEX.md

@@ -8,6 +8,7 @@
 | [Elite TOGA 2.0 → TOGaDeskSupport Standalone Attachment Sync](features/elite-togadesk-attachment-sync.md) | `sync_togadesk_elite_attachments.php` is a standalone cron (every 5 minutes) that syncs file attachments from TOGA 2.0 into TOGaDeskSupport for Elite. | worker/crons/toga2/elite/sync_togadesk_elite_attachments.php, worker/crons/toga2/elite/test_sync_togadesk_elite_attachments.php |
 | [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/netsuite/rest.php, library/app/systemmonitor/netsuiteintegration.php |
+| [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2 platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went dow | worker/.ebextensions/040_disk_memory_hygiene.config, worker/.ebextensions/045_oneuptime_agent.config, worker/ebs/cron.worker.php, worker/ebs/mount-s3fs-folders.php, worker/ebs/apache_settings.php, worker/ebs/setup_phpini.php |
 | [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Diagnosing frozen 1.0 worker cron check-ins (Sentry "missed" flood)](workflows/diagnosing-frozen-cron-checkins.md) | When 1.0 worker cron timestamps freeze and Sentry project `worker1` fills with **`missed`** check-ins, the intuitive diagnosis — a wedged `App_Framework::isProc | worker/.ebextensions/cron.config, library/app/worker.php |

package/knowledge/1.0/apps/worker/features/oneuptime-server-monitor-host-hygiene.md

@@ -0,0 +1,204 @@
+---
+title: OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host
+framework: "1.0"
+repo: worker
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-08-03
+owners: ["jcardinal"]
+files:
+  - worker/.ebextensions/040_disk_memory_hygiene.config
+  - worker/.ebextensions/045_oneuptime_agent.config
+  - worker/ebs/cron.worker.php
+  - worker/ebs/mount-s3fs-folders.php
+  - worker/ebs/apache_settings.php
+  - worker/ebs/setup_phpini.php
+related: ["oneuptime-worker-uptime-monitoring"]
+---
+
+## Summary
+
+The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2
+platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went down from the EC2
+instance running **out of memory and out of disk**. Two things address it:
+
+1. **Host-level observability** — a OneUptime **Server monitor** per worker role, fed by the
+   OneUptime **Infrastructure Agent** running on the box (CPU / memory / disk / network /
+   process metrics every 30s). This is *not* a heartbeat — see
+   [[oneuptime-worker-uptime-monitoring]] for the separate Incoming-Request heartbeat layer.
+2. **Disk/memory hygiene** — a `.ebextensions` config that reaps stale files and rotates
+   logs (safe/additive), plus a set of **behavior-changing fixes that are documented but
+   NOT applied** (Apache prefork sizing, PHP `memory_limit`, s3fs `ensure_diskfree`,
+   cron `flock`).
+
+## Key files / entry points
+
+| File | Purpose | Risk |
+|---|---|---|
+| `.ebextensions/040_disk_memory_hygiene.config` | Deletes stale junk (s3fs `/tmp` cache, `/var/www/cache` staging, orphaned `WORKER_ERROR_*`, deploy litter) + rotates logs | Safe / additive — deletes only stale files |
+| `.ebextensions/045_oneuptime_agent.config` | Installs + supervises the OneUptime infrastructure-agent (static Go binary, v11.7.4) | Safe / additive — no app behavior change |
+
+## How the Server monitor works
+
+- The agent POSTs metrics every **30s** to
+  `{oneuptime_url}/server-monitor/response/ingest/{secret_key}`; OneUptime evaluates the
+  monitor's **Criteria** against them to decide Operational / Degraded / Offline.
+- **AL1 has no systemd**, so `045_…` deliberately skips the agent's `configure`/`start`
+  subcommands (they register a systemd/SysV service and fail here). It writes
+  `/etc/oneuptime-infrastructure-agent/config.json` directly and runs the foreground **`run`**
+  subcommand under a **1-minute cron watchdog** — the same systemd-bypass the ODP Voice
+  container monitor uses.
+- Metrics reported: CPU (aggregate + per-core + user/system/idle/iowait/steal breakdown),
+  load average 1/5/15, memory + swap, **disk per mount** (total/used/free/%, device, fstype,
+  I/O counters — criteria target a specific path), network (aggregate + per-interface bytes/
+  packets/errors/drops, TCP connection counts; cumulative since boot, OneUptime derives
+  throughput from deltas), host info (platform/kernel/uptime/virtualization), and
+  **per-process** pid/name/command/cpu%/mem%/status/threads/user (enables
+  "process is executing" criteria).
+
+## Provisioning — already done, per-role model
+
+`agilant-worker` is one EB Worker tier whose instances each claim ONE stable role
+(`ebs/cron.worker.php`), so there is **one Server monitor per role (7)**, grouped under the
+OneUptime label **`Worker 1.0`**. Each monitor already carries the criteria below, and its
+secret is **hardcoded in `045_oneuptime_agent.config`**, selected at runtime by the role in
+`/etc/worker-role`. Nothing to create — you only need to deploy.
+
+| Role | Monitor name |
+|------|--------------|
+| notification | Notification Worker 1.0 - System Health |
+| database | Database Worker 1.0 - System Health |
+| infrastructure | Infrastructure Worker 1.0 - System Health |
+| toga | TOGa Worker 1.0 - System Health |
+| togadesk | TOGa Desk Worker 1.0 - System Health |
+| sync | Sync Worker 1.0 - System Health |
+| catalog | Catalog Worker 1.0 - System Health |
+
+**Key rotation:** regenerate that monitor's secret in OneUptime and update the matching
+`case` arm in `045_…`. Instance-ids churn but roles are stable, so a recycled box
+re-reports to the same role monitor automatically.
+
+**Creating a monitor from scratch** (only if a new role is added): Monitors → Create Monitor
+→ type **Server / VM** → the secret key appears on the monitor's Settings tab after save.
+Via REST it is `POST /api/monitor` with `{"data":{"projectId":…,"name":…,"monitorType":"Server"}}`
+and an `ApiKey` header — but the secret is server-generated and **not** returned in the
+create body; read it from Settings.
+
+**Verify after deploy**, on the instance:
+
+```bash
+pgrep -fa oneuptime-infrastructure-agent      # process running
+tail -n 40 /var/log/oneuptime-agent.log       # agent log
+```
+
+Last Ping updates within ~1 min.
+
+## Criteria (order matters — first match wins, catch-all Operational last)
+
+| Order | Result state | Check | Condition | Threshold |
+|---|---|---|---|---|
+| 1 | Offline | CPU Usage % | Greater Than | 90 |
+| 2 | Offline | Memory Usage % | Greater Than | 90 |
+| 3 | Degraded | Disk Usage % (path `/`) | Greater Than | 90 |
+| 4 | Degraded | CPU Usage % | Greater Than | 75 |
+| 5 | Degraded | Memory Usage % | Greater Than | 75 |
+| 6 | Operational (fallback) | CPU Usage % | Less Than Or Equal To | 100 |
+
+Each criterion's filter condition is **All**. Recommended additions for this worker
+specifically: Degraded if disk `/` > **80%** (earlier warning — disk is this box's actual
+failure mode), Degraded if swap % used > 50% (early OOM signal once swap exists), and
+Offline if the Apache process is not executing (Server Process → Command contains `httpd`
+→ Is Not Executing).
+
+Via API, criteria live in `monitorSteps.monitorCriteria.criteriaInstances[]`; each instance
+has `filters[]` (`checkOn`, operator, `value`, plus `serverMonitorOptions.diskPath` for disk
+checks), a `filterCondition` (`All`/`Any`), and a `monitorStatusId`.
+
+## Root cause of the outages
+
+### Disk
+
+| Cause | Evidence | Status |
+|---|---|---|
+| s3fs `use_cache=/tmp` **unbounded**, never evicted | `ebs/mount-s3fs-folders.php:53`, `ebs/s3fs.json` (3 buckets) | **Not fixed** — needs `ensure_diskfree` (below) |
+| `/var/www/cache` 0777 with no reaper; DB dumps up to **12 GB/part**, deleted only on successful upload | `ebs/setup_export_cache_folders.php:11-13`, `crons/database/database_backup.php:75,314,337` | Reaper cron shipped in `040_…` |
+| `WORKER_ERROR_*` reaper never scheduled | `crons/worker/check_worker_errors.php` — present in **no** `schedules/*.json` | Reaper cron shipped in `040_…` |
+| Deploy litter (`/tmp/phpredis*`, s3fs source) | `010_setup_redis.config:3-6`, `006_mount-s3fs.config:16-29` | One-off cleanup shipped in `040_…` |
+
+### Memory
+
+| Cause | Evidence | Status |
+|---|---|---|
+| Apache prefork massively over-provisioned: `MaxRequestWorkers = ceil(3200*3/37) = 260` while PHP `memory_limit = 2G` | `ebs/apache_settings.php:3,8` + `ebs/setup_phpini.php:22` | **Not fixed** |
+| Long CLI cron jobs overlap (minute-ly schedules, no `flock`), each entitled to 2 GB | `ebs/cron.worker.php`, `schedules/cron.worker*.json` (26–103 jobs) | **Not fixed** |
+| Standing band-aid: Apache `graceful` every 20 min "to close db connections" | `schedules/cron.worker.json:3-7` | **Not fixed** — would be retired by `MaxConnectionsPerChild` |
+
+## Fixes documented but NOT applied — review before applying
+
+These change app behavior. Apply deliberately.
+
+**Bound the s3fs cache** — `ebs/mount-s3fs-folders.php`, after the `use_cache` option is
+appended (line 53):
+
+```php
+$mount_command .= ' -o ' . escapeshellarg('ensure_diskfree=2000'); // MB; s3fs evicts its own cache to keep >=2GB free
+```
+
+**Right-size Apache** — `ebs/apache_settings.php`: measure the real child size (the command
+is in the file at line 4 — with `memory_limit=2G`, heavy endpoints are far bigger than the
+assumed 37 MB), then replace the `× 3` overcommit on line 8 with
+`MaxRequestWorkers = floor(availableRAM_MB * 0.7 / measuredChildMB)` (≈20–25 on a 4 GB box,
+not 260) and lower `StartServers`/`MinSpareServers` to 4–8. Add `MaxConnectionsPerChild 500`
+inside the `<IfModule prefork.c>` block so children recycle and return leaked memory to the
+OS — this is what retires the 20-minute graceful-restart hack.
+
+**Lower cron memory pressure** — `ebs/setup_phpini.php`: drop the global
+`memory_limit = 2G` (line 22) to `512M` and raise it per-script via `ini_set()` only in the
+few CLI batch jobs that need it; set `display_errors = off` (line 20) in production, logging
+to a rotated file instead.
+
+**Stop cron overlap** — where `ebs/cron.worker.php` builds each cron command, prefix
+minute-ly/2-minute jobs with a non-blocking lock:
+`flock -n /tmp/lock.<jobname> php /var/www/html/crons/<jobname>`.
+
+**Swap (optional stopgap)** — only *after* the disk reapers are in place; this box is
+disk-constrained and a 2 GB swapfile on a near-full root volume makes disk worse:
+
+```yaml
+commands:
+  02_swap:
+    test: "[ ! -f /var/swapfile ]"
+    command: "dd if=/dev/zero of=/var/swapfile bs=1M count=2048 && chmod 600 /var/swapfile && mkswap /var/swapfile && swapon /var/swapfile"
+```
+
+## Gotchas
+
+- **Disk path `/` works here** — this is a real EC2 host, unlike the ODP Voice container
+  case where "Disk metrics for path / were unavailable" appears because the container root
+  isn't in the reported disk list. s3fs (fuse) mounts may **not** be reported as normal
+  partitions, so prefer alerting on `/` rather than the s3fs mount.
+- The agent is kept alive by a **cron watchdog, not a service** — if it stops reporting,
+  check the watchdog and `/var/log/oneuptime-agent.log`, not `service`/`systemctl`.
+- Monitor secrets are hardcoded per-role in `045_…`; treat that file as a credential
+  location. Do not copy secrets into docs, logs, or other files.
+- **Security follow-up (urgent, out of scope of the monitoring work):** hardcoded live AWS
+  credentials and DB passwords are committed in plaintext in `ebs/cron.worker.php:14-15`,
+  `crons/worker/worker_heartbeat.php:12`, `crons/database/database_backup*.php`, and
+  `config.worker.ini`. Rotate and move to EB environment properties / an IAM instance role.
+
+## References
+
+- Agent + subcommands — https://github.com/OneUptime/oneuptime/blob/master/InfrastructureAgent/README.md
+- Server monitor — https://oneuptime.com/docs/monitor/server-monitor
+- Criteria instance API — https://oneuptime.com/reference/monitor-criteria-instance
+- Monitor REST API — https://oneuptime.com/reference/monitor
+- Agent releases (v11.7.4) — https://github.com/OneUptime/oneuptime/releases
+
+## Change history
+- 2026-08-03 — Captured from `worker/docs/oneuptime-server-monitor.md` (that in-repo doc is
+  being deleted; this is now the source of truth). Documents the per-role OneUptime Server
+  monitors + infrastructure agent (`045_…`), the shipped disk/memory hygiene reapers
+  (`040_…`), the diagnosed disk/memory root causes, and the reviewed-but-unapplied
+  behavior-changing fixes. (jcardinal)

package/knowledge/1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md

@@ -11,7 +11,7 @@ owners: ["jcardinal"]
 files:
   - library/app/worker.php
   - worker/crons/worker/worker_heartbeat.php
-related: []
+related: ["oneuptime-server-monitor-host-hygiene"]
 ---
 
 ## Summary

package/knowledge/1.0/standards/backend-php.md

@@ -5,11 +5,12 @@ project: Library
 client: shared
 type: standard
 status: active
-updated: 2026-07-24
+updated: 2026-08-04
 owners: [jcardinal, rgirish, mhammontree]
 files: []
 related:
   - ../apps/library/architecture.md
+  - ../apps/library/features/error-capture-1-0.md
   - ../../2.0/standards/backend-php.md
 ---
 
@@ -259,6 +260,49 @@ if ($condition) {
 $piece = substr($value, 0, 8);
 ```
 
+### No typed properties (PHP 7.2)
+
+* 1.0 runs **PHP 7.2**, so **typed properties (7.4+) are forbidden anywhere in `library/` or a 1.0
+  app.** Declare class attributes untyped with an `@var` docblock:
+
+```php
+// WRONG in 1.0 — deploy fails with "syntax error, unexpected '?'"
+private static ?string $memoryReserve = null;
+
+// CORRECT in 1.0
+/** @var string|null */
+private static $memoryReserve = null;
+```
+
+* **Parameter and return types ARE fine** (PHP 7.0/7.1) — this rule is only about *properties*.
+* This is the opposite of the 2.0 standard, which requires typed properties. The universal
+  `rules/toga/common/coding-style.md` rule is already **scoped to 2.0** and cross-references this
+  standard for the 1.0 `@var` docblock form — so there is no conflict to resolve, just two different
+  rules for two different runtimes.
+* Real failure: a deploy of `library/app/error/capture.php` broke on
+  `private static ?string $memoryReserve`; nine typed properties across three files had to be
+  converted.
+
+### Any PHP warning terminates the request — how to opt out locally
+
+`App_Error::handleError()` routes **every** PHP warning into `handleException()`, which renders the
+error page and calls **`exit()`**. (2.0's equivalent *throws*, which a `catch` can absorb — 1.0 does
+not give you that.) So a warning raised inside code that must not escalate — error capture, a
+monitoring ping, an IMDS probe where `file_get_contents` warns before returning `false`, a retry
+loop — **kills the request or cron mid-run while reporting an unrelated error.**
+
+Two sanctioned patterns, in order of preference:
+
+1. **A local handler around the risky call** — `set_error_handler()` throwing an `ErrorException`,
+   then `restore_error_handler()`. Use this when you want to *catch and retry* (the NetSuite
+   SSL-drop retry loop does this).
+2. **`App_Error::setThrowExceptionsEnabled(false)` with a `finally` restore** — use this when the
+   code must never escalate at all (`App_Error_Capture::captureException()` does this for its whole
+   duration). This disables the **handler**, not real `throw`s: `App_Query` raises a plain
+   `throw new Exception` on a DB error, so genuine failures still reach your `catch`.
+
+Never leave either one in effect beyond the block that needs it.
+
 ### Documentation
 
 * **DocBlocks:** framework (`library`) classes/methods are well documented with a description and `@param`/`@return`/`@throws`. Application code is less consistent — document new and modified methods to the framework standard.

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -17,7 +17,7 @@
 | [Re-pointing a DB alias mid-request (_Database::register park/restore)](features/database-alias-repointing.md) | `_Database` keys **all live per-database runtime state by the connection ALIAS** (`Client` / `_underscore::DB_CLIENT`, `ClientLogs`, `Archive`), **not** by the | _underscore/Database.php, _underscore/Query.php, api2/Component/Api/V2/V2.php, api2/Component/Api/CrossClient/CrossClient.php |
 | [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. | _underscore/Email.php, worker2/Worker/Infrastructure/Email/Send.php |
 | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
-| [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Exception/Business.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Logs/2026-08-01a - Rename tables to singular.sql, dbchanges2/Logs_Client/2026-08-01a - Drop Error table.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
+| [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Database.php, _underscore/Exception/Business.php, api2/Controller/Index.php, worker2/Controller/Index.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Logs/2026-08-01a - Rename tables to singular.sql, dbchanges2/Logs_Client/2026-08-01a - Drop Error table.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
 | [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
 | [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | worker2/Component/Forecast/SaleImport/SaleImport.php, worker2/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/fixer.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
 | [isFulfillable Propagation Up the SO↔PO Chain](features/fulfillable-item-propagation.md) | `Items.isFulfillable` is a boolean that gates whether a storefront line's **Qty Fulfilled** cell is actionable. | _underscore/Model/Client/Item.php, _underscore/Model/Compass/Item.php, dbchanges2/Core/2026-07-17 - Items isFulfillable RecordField.sql, dbchanges2/Core/2026-07-17 - RegisterItemIsFulfillableInterceptors.sql, dbchanges2/Client/2026-07-17 - ItemsisFulfillable.sql |