@remits/remits-cli 0.1.113 → 0.1.115

package/skills/remits-cli/references/investigation.md

@@ -0,0 +1,254 @@
+# Investigation and Production Support
+
+> A `remits-cli` skill reference. **Load this when** you are investigating live behavior: which record trail to read, how to correlate a processing chain, and the production support flows.
+>
+> The table of contents below carries **real line numbers** (`- L84  Some Heading`), resolved when
+> this file is installed, so they are never stale. Read the head, pick your sections, and offset-read
+> only those. The entry text is the heading verbatim, so it also greps.
+
+## Table of Contents
+
+- [The Investigation Model](#the-investigation-model)
+  - [Which tool reads which record](#which-tool-reads-which-record)
+  - [Correlation keys](#correlation-keys)
+  - [Reading a record's `content` — persisted context, not a memory dump](#reading-a-records-content--persisted-context-not-a-memory-dump)
+  - [HTTP audits](#http-audits)
+  - [AI activity](#ai-activity)
+  - [Node Reference Table](#node-reference-table)
+  - [Runtime node and `localMode`](#runtime-node-and-localmode)
+- [Production Support Workflow](#production-support-workflow)
+  - [Investigation Strategy](#investigation-strategy)
+  - [Presenting Findings](#presenting-findings)
+  - [Verifying a Production Issue Fix](#verifying-a-production-issue-fix)
+
+## The Investigation Model
+
+Front stage — Schemas, Readers, Actions, Embeddables, Rules, HtmlTemplates, Agents, Tests, and the Firestore
+**documents** that hold business data — is described in `platform-overview.md`. This section is only about
+the **back-stage lifecycle records** an investigation actually reads, and how to read them.
+
+### Which tool reads which record
+
+The model — Documents (Firestore business data) vs Objects and their record trail (MySQL), linked by
+`object_id` — is in `platform-overview.md` → *Objects, Documents, and the Record Trail*. What matters
+here is the tool and the filterable fields:
+
+| Record | Query it with | Fields worth filtering on |
+|---|---|---|
+| document | `mcp_firestore_search` | any schema field, plus `account_id`, `object_id`, `_lastModifiedAt` |
+| `object` | `mcp_record_listing` / `mcp_record_view` | `status`, `type`, `name`, `referenceId` |
+| `object_log` | `mcp_record_listing` / `mcp_record_view` | `type`, `description`, `content`, `threadGroupingId` |
+| `event` | `mcp_record_listing` / `mcp_record_view` | `action`, `status`, `eventDate`, `threadGroupingId` |
+| `alert` | `mcp_record_listing` / `mcp_record_view` | `status`, `type`, `active`, `threadGroupingId` |
+| user activity session | `mcp_user_activity` | `userId`, `accountId`, `sessionKey`, `focusedOnly`, `traceId` pivots |
+
+`mcp_record_listing` finds candidates when you do not know the id; `mcp_record_view` opens an exact one;
+`mcp_object_activity` returns one Object's whole timeline in order.
+
+**Check the data lane before concluding a record does not exist** — `--data-mode test` and `prod` read
+different lanes (see `platform-overview.md` → *Test and Production Data Lanes*).
+
+### Correlation keys
+
+- **`object_id`** — links documents to records. Pivot `mcp_firestore_search` → `mcp_object_activity`.
+- **`threadGroupingId`** — groups every record and log line from one processing chain, and is also the
+  request's trace id. Pivot into `mcp_system_logs` and `mcp_performance_trace`.
+- **`node`** — resolves to `serviceName` + `region` for log queries (table below).
+- **`sessionKey`** — salted user-activity session key. Pivot `mcp_user_activity` `sessions` → `story`;
+  each beat then carries a `traceId` / `threadGroupingId` for trace and log investigation.
+- **`sessionId`** — pivots into persisted AI activity via `mcp_ai_session_search`.
+
+### Reading a record's `content` — persisted context, not a memory dump
+
+`content` on an `object`, `object_log`, `event`, or `alert` is the **sanitized snapshot** the platform
+deliberately kept of what the producing component knew at that workflow step. It is not everything that was
+in memory.
+
+- `Object.content` — ingestion/request/file context when the Object was created or updated.
+- `ObjectLog.content` — the logging component's point-in-time view.
+- `Event.content` — the scheduling component's view, plus the explicit event options.
+- `Alert.content` — the raising component's view, plus the explicit alert body.
+
+**Missing or `[REDACTED]` does not mean the component never had it.** Before persisting, the platform drops
+non-serializable objects, strips `requestBody` / `params` / `token` recursively, summarizes very large
+strings, prunes oversized maps and collections, and redacts secret-looking keys (`password`, `secret`,
+`authorization`, `cookie`, `clientSecret`, …) plus any Account/User schema field marked `sensitive: true`.
+When explaining a record, distinguish what the workflow *had* at runtime from what the platform *kept*.
+
+**Provenance travels in the context itself:**
+
+- **`source_bcd`** — the immediate component that produced this record, e.g. `[Action:18] DataPlus Invoice Posting`
+- **`upstream_source_bcd`** — the prior workflow hop, when the context was inherited from one
+
+**Never interpret `content` in isolation.** Pair it with the record type, `source_bcd`, any
+`upstream_source_bcd`, the `object_id` timeline, the `threadGroupingId` chain, and the producing
+component's source. The goal is not to find a suspicious record — it is to explain **why the context looks
+exactly the way it does relative to the workflow step that produced it**.
+
+### HTTP audits
+
+Raw inbound (Reader) and outbound (`rest(...)`) HTTP is persisted to Firestore — but **only for accounts
+with `Account.enableHttpAudits`**. When it is off, the absence of audit documents proves nothing.
+
+Audits live in **monthly** collections, not one global collection:
+
+```
+http-audits/http-audits-YYYY-MM/entries
+```
+
+Start with the month the request ran in, and check the adjacent month if the run may have crossed a
+boundary. Query them with `mcp_firestore_search` like any other collection; the filters worth reaching for
+are `direction` (`INBOUND` / `OUTBOUND`), `success`, `request.method`, `request.path`, `component.name`, and
+`response.statusCode`. Some paths are deliberately excluded from persistence, so a missing audit is not
+proof a call was never made.
+
+Reach for audits when the question is *what exact request went out, what came back, and did this component
+actually make the call* — then use the component source plus the payload to place the fault in request
+formation, the partner's response, or downstream processing. Full captured shape, websocket behavior, and
+embeddable patterns: `features/http-audits.md` (`mcp_get_guide`).
+
+### AI activity
+
+Every `ai()` call and Agent turn is persisted. Two ways in:
+
+- **`mcp_ai_session_search`** — search groupings and open their detail. A grouping spans every session
+  sharing one grouping id: the agent turns plus its guardrail and internal `ai()` calls. Use the
+  **map → open** flow described under its `tool-reference.md` entry, never a whole-detail dump.
+- **`ai_request_response([sessionId: id])`** — from inside component code, loads the stored
+  request/response history for a session (`first: true` / `last: true` for one record). This is also how
+  you replay a stored provider response in a test.
+
+If a document or agent state already carries a session id, that is a direct pivot into persisted AI
+activity.
+
+**Before tuning any prompt, read the session.** `features/ai-session-investigation.md` owns the method —
+per-turn forensics, what each layer proves, and the rule that what a tool **PRODUCED** is not necessarily
+what the model **CONSUMED**. `features/ai-strategy.md` owns what to change once you know. Changing a prompt
+before reading the persisted request/response is guessing.
+
+### Node Reference Table
+
+| Node Name | Service Name | Region |
+|---|---|---|
+| remitsAdmin-east5 | remits | us-east5 |
+| remitsActions | remits-actions | us-east1 |
+| remitsAdmin | remits | us-east1 |
+
+`mcp_system_logs` accepts `node` directly and resolves it automatically.
+
+### Runtime node and `localMode`
+
+The deployed `remits` service in `us-east5` (`remitsAdmin-east5`) runs with the platform setting
+`localMode=true`. If someone says "localModel" in this context, confirm they mean this `localMode`
+setting. Operationally, immediate async follow-on work stays on the same Cloud Run service/node instead of
+being sharded to `remits-actions`:
+
+- Pub/Sub-style follow-on messages are handled locally after commit.
+- Near-immediate tasks are handled locally when `localMode` is enabled. Future scheduled tasks still use
+  Cloud Tasks.
+- Local worker hops preserve the run context, including staged-source resolution, data mode,
+  `threadGroupingId`, and trace correlation.
+- Durable boundaries such as async HTTP ingress and Events carry that same run context across the queue.
+
+For investigations on the default deployed host (`https://remits-529558023549.us-east5.run.app`), do not
+assume "async" means `remitsActions` / `us-east1`. Start with `node:"remitsAdmin-east5"` and the
+`threadGroupingId`; pivot to `remitsActions` only when the Event delivery envelope, log line, or returned
+node says the work actually ran there.
+
+This does not change the data-lane rule: a non-null `TestMode` can exist only to carry branch/staged-source
+resolution. Data isolation is decided by CLI `--data-mode`: a branch-scoped `--data-mode prod` run is still
+prod data, while `--data-mode test` remains isolated test data.
+
+## Production Support Workflow
+
+Switch to prod mode for investigations:
+
+```bash
+remits-cli data-mode set prod
+```
+
+### Investigation Strategy
+
+Before starting an investigation outside the confirmed current repo:
+1. Read `~/.remits-cli/account-repos.json`
+2. Switch to the best local repo candidate
+3. Read that repo's `account-info.json`
+4. Confirm whether you are in `CLIENT`, `PLATFORM`, or `PRODUCT` context
+5. Then continue with the investigation flow below
+
+**Document-First** (most common — user reports a data issue):
+1. `mcp_account_view` — understand the account's schemas and components.
+2. `mcp_firestore_search` — find the document, capture its `object_id`.
+3. If the issue involves inbound or outbound HTTP behavior and the account has `enableHttpAudits`, query `http-audits/http-audits-YYYY-MM/entries` with `mcp_firestore_search`.
+4. `mcp_object_activity` — scan the timeline for warnings, errors, unexpected events.
+5. `mcp_record_listing` — search or filter alerts, events, object logs, or objects when you need to find the suspicious record first.
+6. `mcp_record_view` — drill into suspicious entries for full content.
+7. `mcp_ai_session_search` — if the workflow involves AI, inspect session groupings, prompts, tool definitions, and responses in human-readable form.
+8. `mcp_user_activity` — for "user X is slow right now" reports, list sessions by `userId`/`accountId`, open the session story, and use the returned beat pivots.
+9. `mcp_performance_trace` — for slow/sluggish reports, open the beat `traceId` with `action:"trace"`; use `action:"slowest"` when you only have a broad time window.
+10. `mcp_system_logs` — correlate via `threadGroupingId` for raw log context when the trace needs supporting log lines.
+11. If no local checkout exists for the responsible implementation account, use
+    `mcp_component_view`/`mcp_component_grep` to explain how the component works. If the repo exists on
+    this machine, inspect the branch/files there instead; the remote component tools are fallback and
+    live-DB comparison surfaces, not the starting point for source comprehension.
+
+**Slow / sluggish user report:**
+
+1. Resolve the reporting user/account with `mcp_account_user_admin` if you only have an email/name.
+2. Call `mcp_user_activity` with `action:"sessions"` and `userId` or `accountId`.
+3. Open the likely row with `action:"story"` and inspect beat labels, status, `ms`, `node`, and `traceId`.
+4. Open slow or failed beat pivots with `mcp_performance_trace` before querying raw logs.
+5. Use the returned `mcp_system_logs` pivot only when the trace needs surrounding log lines.
+6. If there is no live session, call `mcp_user_activity` `action:"watch"` for the user/account, ask for reproduction, then read `sessions`/`story` again. Focused sessions retain sanitized request detail and emit archived `REMITS_ACTIVITY` log lines.
+
+**Error or Alert Investigation:**
+1. `mcp_record_listing` — search by alert type, content, error text, action, status, `threadGroupingId`, or other exact-match record properties when you do not yet know the record ID.
+2. `mcp_record_view` — inspect the chosen record/event/alert/object in full once you have its ID.
+3. Use `object_id` + `threadGroupingId` to pull full timeline and logs.
+4. Cross-check Firestore document state.
+5. Identify `source_bcd` and any `upstream_source_bcd`.
+6. Explain the record in terms of the workflow step that produced it, not as a generic JSON blob.
+7. If the context looks missing, redacted, or truncated, consider sanitization rules before concluding data was never present.
+8. If AI behavior is part of the symptom, use `mcp_ai_session_search` and compare the persisted session content against the Agent component implementation and `features/ai-support.md`.
+
+**Stuck / failed / recovered Event:**
+
+Do **not** open the Action source first. The platform records each attempt's delivery envelope — which
+queue delivered it, which delivery attempt this was, and how long it was ever allowed to run — and
+classifies the failure for you.
+
+```bash
+remits-cli tool --name mcp_event_diagnostics --input '{"accountId":49,"eventId":18838}' --data-mode prod
+```
+
+The same classifier is available from a Test or any component as `eventDiagnostics(18838)`.
+
+**Read `classification` before anything else** — only `APPLICATION_FAILURE` means the bug is in the
+component. The full classification table, what each `abandonmentCause` implies, and the returned `pivots`
+are under **`mcp_event_diagnostics`** in `tool-reference.md`. One thing to check every time:
+`delivery.deliveryAttempt` above `1` means Cloud Tasks had **already** retried this event, so any
+non-idempotent side effect may have run more than once — look for duplicate records before concluding the
+component "ran twice for no reason".
+
+Full detail: `features/observability.md` and `features/events-builder-guide.md` (`mcp_get_guide`).
+
+### Presenting Findings
+
+Users are not engineers. When reporting investigation results:
+- Lead with what happened in plain language.
+- Show the evidence (document values, timeline events, log excerpts).
+- Explain why it happened if you can determine the cause.
+- Recommend what to do next — in terms the user can act on.
+
+### Verifying a Production Issue Fix
+
+When a bug is reported from production, use this pattern:
+
+1. Investigate the live issue in **prod mode** and identify the exact affected document IDs, collection names, account IDs, and component path.
+2. Make the code change in the owning `PLATFORM` or `PRODUCT` repo when the defect is in shared implementation.
+3. Verify in **test mode**, not prod.
+4. Prefer a **Test component** when the behavior can be asserted programmatically, because that creates a durable regression suite and lets you explicitly construct the necessary data, operations, and assertions.
+5. Use `remits-cli token` plus `playwright-cli` when the proof is visual or interaction-driven.
+6. If useful, create or update a dedicated embeddable "playground" in test mode to reproduce the scenario in a controlled way.
+
+Do not move production customer data into another account's test collection as a routine verification strategy. If you cannot verify with a Test component, Playwright flow, or controlled test-mode embeddable, explain the gap clearly instead of improvising with live production validation.