@raishin/vanguard-frontier-agentic 2.3.0 → 2.5.0

package/skills/salesforce/salesforce-agentforce-stdm-observer-skill/references/observability-rubric.md

@@ -0,0 +1,219 @@
+# Observability Rubric
+
+Metric thresholds, interpretation guidance, and escalation matrix for
+`salesforce-agentforce-stdm-observer-skill`.
+
+**Note:** Salesforce does not publish official fixed thresholds
+for all STDM metrics. Where official thresholds exist, they are cited. Where
+they do not, operationally reasonable defaults are provided with a
+`[PLACEHOLDER]` annotation. Validate against your
+org's baseline before treating any threshold as authoritative.
+
+---
+
+## Faithfulness
+
+Faithfulness measures whether the agent's response is grounded in retrieved
+content rather than hallucinated. A faithfulness score of 1.0 means the
+response was fully supported by retrieved knowledge; 0.0 means none of it was.
+
+| Score range | Interpretation | Action |
+|---|---|---|
+| 0.90 – 1.00 | Excellent — responses strongly grounded | No action required |
+| 0.80 – 0.89 | Acceptable — minor grounding gaps | Monitor trend; investigate individual low-score moments |
+| 0.70 – 0.79 | Warning — meaningful hallucination risk | Hand off to `salesforce-agentforce-risk-review-skill` |
+| < 0.70 | Critical — high hallucination rate | Immediate hand off; flag for human review of agent knowledge base |
+
+**Official reference:** Salesforce STDM `avg_faithfulness` is derived from
+the `AiRetrieverQualityMetric` DMO. The 0.8 threshold for the `Hallucination`
+observability query type is set in `AgentforceOptimizeService`. A threshold
+of 0.80 for the `Hallucination` query type is the value used by the
+forcedotcom/sf-skills `observing-agentforce` reference implementation. Treat
+it as a reasonable default; calibrate to your org's baseline.
+
+**Escalation trigger:** `avg_faithfulness` < 0.80 → hand off to
+`salesforce-agentforce-risk-review-skill` with `anomalies_detected` payload.
+
+---
+
+## Answer Relevance
+
+Answer relevance measures whether the agent's response actually addresses the
+user's question. A score of 1.0 means fully relevant; 0.0 means unrelated.
+
+| Score range | Interpretation | Action |
+|---|---|---|
+| 0.85 – 1.00 | Excellent — responses on-topic | No action required |
+| 0.70 – 0.84 | Acceptable — occasional off-topic responses | Monitor; review top intents vs. agent topics |
+| 0.60 – 0.69 | Warning — frequent off-topic responses | Hand off to `salesforce-agentforce-risk-review-skill` |
+| < 0.60 | Critical — agent consistently off-topic | Immediate hand off; likely topic routing configuration issue |
+
+**Official reference:** The `AnswerRelevancy` query type in
+`AgentforceOptimizeService` flags subagents with `avg_answer_relevance < 0.7`.
+[PLACEHOLDER: Salesforce has not published a canonical
+"good" answer relevance target in its public documentation as of 2026-05-21.
+The 0.70 threshold in the sf-skills observability query is used here as the
+lower bound; 0.85 as the "acceptable" floor reflects operationally reasonable
+calibration.]
+
+**Escalation trigger:** `avg_answer_relevance` < 0.70 → hand off to
+`salesforce-agentforce-risk-review-skill`.
+
+---
+
+## Context Precision
+
+Context precision measures the quality of RAG retrieval — whether the
+retrieved knowledge chunks were relevant to the query.
+
+| Score range | Interpretation | Action |
+|---|---|---|
+| 0.85 – 1.00 | Excellent — retrieval highly targeted | No action required |
+| 0.70 – 0.84 | Acceptable — some irrelevant chunks retrieved | Monitor; may indicate knowledge base over-breadth |
+| 0.60 – 0.69 | Warning — retrieval quality degraded | Hand off; review knowledge base indexing and retrieval config |
+| < 0.60 | Critical — retrieval largely irrelevant | Immediate hand off; agent responses likely grounded on wrong content |
+
+[PLACEHOLDER: Salesforce STDM documentation does not
+specify canonical context precision thresholds. These values are adapted from
+RAG evaluation literature and the sf-skills `KnowledgeGap` query type which
+reports the lowest-scoring retrievers first.]
+
+**Escalation trigger:** `avg_context_precision` < 0.70 → include in
+`anomalies_detected`; hand off to `salesforce-agentforce-risk-review-skill`
+with `KnowledgeGap` observability query results.
+
+---
+
+## Quality Score (1–5 Scale)
+
+The STDM quality score is a 1–5 integer assigned per moment by an LLM
+evaluator. It maps to UI labels in Agentforce: 5=High, 3-4=Medium, 2=Low,
+1=Very Low.
+
+| avg_quality_score | Interpretation | Action |
+|---|---|---|
+| 4.5 – 5.0 | Excellent | No action required |
+| 4.0 – 4.4 | Good | Monitor |
+| 3.0 – 3.9 | Moderate — investigate low-scoring moments | Review quality_distribution; identify dominant failure mode |
+| 2.0 – 2.9 | Poor | Hand off to `salesforce-agentforce-risk-review-skill` |
+| < 2.0 | Critical | Immediate escalation; agent may be actively harming user experience |
+
+**Quality distribution signal:** A bimodal distribution (many 5s and many 1s)
+suggests inconsistent behavior that is harder to diagnose than a uniformly
+low score. Both patterns warrant escalation.
+
+**Escalation trigger:** `avg_quality_score` < 4.0 → include in
+`anomalies_detected`; investigate quality_distribution for dominant failure
+category.
+
+---
+
+## Action Invocation Error Rate
+
+The action error rate is computed as:
+`error_rate = action_error_count / action_invocation_count`
+
+| Error rate | Interpretation | Severity | Action |
+|---|---|---|---|
+| 0% – 2% | Normal — transient errors within tolerance | Low | No action required |
+| 2% – 5% | Elevated — investigate action reliability | Medium | Monitor trend; review action error messages |
+| 5% – 10% | High — action reliability issue | High | Hand off to `salesforce-agentforce-risk-review-skill` |
+| > 10% | Critical — systemic action failure | Critical | Immediate hand off; agent may be failing to complete tasks |
+
+**Escalation trigger:** `error_rate` > 0.05 → hand off to
+`salesforce-agentforce-risk-review-skill` with error breakdown payload.
+
+---
+
+## Session Abandonment Rate
+
+Abandonment rate = sessions that ended without the agent completing the user's
+intent (proxied by `UNKNOWN` end_type or sessions shorter than a minimum
+meaningful duration).
+
+| Abandonment rate | Interpretation | Action |
+|---|---|---|
+| 0% – 15% | Normal | No action required |
+| 15% – 30% | Elevated | Investigate top intents for dead-ends; check action availability |
+| 30% – 50% | High | Hand off; agent likely has missing actions or broken routing |
+| > 50% | Critical | Immediate escalation; agent not usable in production |
+
+[PLACEHOLDER: Salesforce does not publish canonical
+abandonment thresholds. These ranges are operationally calibrated and should
+be adjusted to org-specific baseline. An org with a narrow, self-service use
+case may have a naturally lower abandonment rate than a broad-intent agent.]
+
+**Escalation trigger:** `abandonment_rate` > 0.30 → include in
+`anomalies_detected` with severity `high`.
+
+---
+
+## Faithfulness Sudden Drop (Regression Detection)
+
+A sudden drop in `avg_faithfulness` between two consecutive observation
+windows (even if the absolute value remains above threshold) is an anomaly
+signal:
+
+| Drop magnitude (vs. prior window) | Severity | Action |
+|---|---|---|
+| < 5% decrease | Normal variation | No action |
+| 5% – 15% decrease | Warning | Note in anomalies; observe next window |
+| > 15% decrease | High | Hand off; potential knowledge base change or agent version drift |
+| Faithfulness drops from > 0.85 to < 0.75 in one window | Critical | Immediate hand off regardless of absolute value |
+
+**Hallucination signature patterns:**
+- `avg_faithfulness` drops sharply while `avg_answer_relevance` remains
+  stable → agent is generating confident but ungrounded responses; likely
+  a retrieval failure or knowledge base gap.
+- Both `avg_faithfulness` and `avg_answer_relevance` drop → agent may have
+  been updated or its topic routing changed; compare against prior config.
+- Quality score distribution shifts toward 1-2 with no change in error rate
+  → likely LLM generation quality degraded (possible model change).
+
+---
+
+## Escalation Matrix
+
+| Condition | Severity | Downstream skill | Payload |
+|---|---|---|---|
+| avg_faithfulness < 0.80 | High | `salesforce-agentforce-risk-review-skill` | audit_envelope, aggregate_metrics, Hallucination observability results |
+| avg_answer_relevance < 0.70 | High | `salesforce-agentforce-risk-review-skill` | audit_envelope, aggregate_metrics, AnswerRelevancy results |
+| avg_context_precision < 0.70 | Medium | `salesforce-agentforce-risk-review-skill` | audit_envelope, KnowledgeGap results |
+| action error_rate > 0.05 | High | `salesforce-agentforce-risk-review-skill` | audit_envelope, error breakdown |
+| action error_rate > 0.10 | Critical | `salesforce-agentforce-risk-review-skill` + notify Live Guard | audit_envelope, error breakdown, change_proposal flag |
+| abandonment_rate > 0.30 | High | `salesforce-agentforce-risk-review-skill` | audit_envelope, aggregate_metrics |
+| avg_quality_score < 4.0 | Medium | `salesforce-agentforce-risk-review-skill` | audit_envelope, quality_distribution |
+| Faithfulness regression > 15% in one window | High | `salesforce-agentforce-risk-review-skill` | audit_envelope, regression signal |
+| A configuration change is proposed | Any | `salesforce-live-guard-agent` | audit_envelope, change_proposal |
+| Regulated-vertical org + any anomaly | Any | `salesforce-compliance-privacy-agent` | audit_envelope, anomalies_detected, vertical_flag |
+
+---
+
+## Composite Health Score
+
+When reporting overall agent health, compute a composite score from
+the component metrics. This is advisory only; the threshold is not authoritative.
+
+```
+composite_health = (
+    (avg_faithfulness     * 0.35) +
+    (avg_answer_relevance * 0.30) +
+    (avg_context_precision * 0.15) +
+    ((avg_quality_score / 5.0) * 0.10) +
+    ((1.0 - error_rate) * 0.05) +
+    ((1.0 - abandonment_rate) * 0.05)
+)
+```
+
+| Composite health | Interpretation |
+|---|---|
+| 0.90 – 1.00 | Excellent |
+| 0.80 – 0.89 | Good |
+| 0.70 – 0.79 | Acceptable — monitoring recommended |
+| 0.60 – 0.69 | Concerning — hand off to risk-review |
+| < 0.60 | Poor — immediate escalation |
+
+[PLACEHOLDER: The composite weighting above is an
+operationally calibrated heuristic. Adjust weights to reflect your org's
+business priorities — e.g., a regulated financial services agent may weight
+faithfulness more heavily than context precision.]

package/skills/salesforce/salesforce-agentforce-stdm-observer-skill/references/privacy-redaction.md

@@ -0,0 +1,240 @@
+# Privacy Redaction Rules
+
+Agentforce-specific redaction policy for `salesforce-agentforce-stdm-observer-skill`.
+
+This skill operates on aggregate metrics only. Session text content — user
+messages, agent responses, LLM prompt/response pairs — is never emitted. This
+is a structural design decision, not a configurable option.
+
+**Verify-before-merge:** Agentforce session data handling requirements and
+applicable data residency rules vary by Salesforce product edition, org region,
+and customer data classification. Validate against your organization's data
+governance policy and the Salesforce Data Processing Addendum before deploying
+this skill in a regulated environment.
+
+---
+
+## Why Session Content Is Never Emitted
+
+Agentforce sessions capture live conversational exchanges between users and the
+agent. That content may include:
+
+- User names, contact information, account details
+- Health, financial, or HR data depending on agent scope
+- Customer complaints, support issues, or sensitive business context
+- Internal record IDs that could be combined with other data to identify individuals
+- In regulated verticals (Health Cloud, Financial Services Cloud): PHI, PII, or
+  regulated financial information
+
+Even a single session that appears to contain only "generic" content cannot be
+safely classified as PII-free without examining it — which creates a circular
+dependency. The aggregate-only policy sidesteps this problem entirely: the skill
+never touches session content, so there is no PII surface to manage in output.
+
+If raw session content is genuinely required (e.g., a developer must debug a
+specific hallucination in a production session), that access must be:
+
+1. Requested through `salesforce-live-guard-agent` for human-in-the-loop
+   confirmation.
+2. Performed by a human operator with appropriate data access permissions,
+   not by an automated agent.
+3. Handled according to the org's data classification and retention policy.
+
+See the Human-in-the-Loop Path section for the routing procedure.
+
+---
+
+## Redaction Rules (Apply in Order)
+
+### Rule 1: Session text content — absolute prohibition
+
+**Never** include any of the following in output, intermediate files, audit
+envelopes, or log entries:
+
+- `ssot__ContentText__c` (AiAgentInteractionMessage — raw user/agent messages)
+- `ssot__InputValueText__c` (AiAgentInteractionStep — action input data)
+- `ssot__OutputValueText__c` (AiAgentInteractionStep — action output data)
+- `ssot__PreStepVariableText__c` / `ssot__PostStepVariableText__c` (variable snapshots)
+- `prompt` / `llm_response` from `getLlmStepDetails` return values
+- `request_summary` / `response_summary` from `getMomentInsights` return values
+  (LLM-synthesized paraphrases — still potentially sensitive)
+- `quality_reasoning` from moment insights (LLM-generated explanation referencing
+  session content)
+
+**What to emit instead:** numeric scores, counts, timestamps, and metric names only.
+
+### Rule 2: OAuth tokens, refresh tokens, session IDs
+
+Never include in any output, log, or audit envelope field. If they appear in
+CLI output, strip before emitting. This applies to:
+
+- OAuth access tokens
+- Refresh tokens
+- `sf` CLI auth credentials
+- Salesforce session cookies
+
+### Rule 3: Salesforce Org IDs
+
+Salesforce Org IDs are 18-character strings starting with `00D`. Replace with
+`<org_id_placeholder>` in all output and audit envelopes.
+
+### Rule 4: Salesforce Record IDs (general)
+
+15-character or 18-character Salesforce record IDs appearing in query results,
+step data, or metadata must not be echoed. Replace with `<record_id_placeholder>`.
+
+**Exception for action invocation deduplication:** When counting unique records
+accessed by action invocations (e.g., to detect an agent repeatedly querying
+the same record), hash the raw record ID to a deterministic 6-character token
+(e.g., `rec_a3f2`) using SHA-256 truncated. This allows duplicate detection
+without exposing the raw ID. Never emit the source ID alongside the hash.
+
+### Rule 5: Session participant IDs and user IDs
+
+Session participant IDs, user IDs (`005...`), and any other identifiers that
+link a session to a specific Salesforce user must be replaced with
+`<user_id_placeholder>`.
+
+This applies to:
+- `ssot__ParticipantId__c` on `AiAgentSessionParticipant__dlm`
+- `OwnerId`, `CreatedById`, `LastModifiedById` on any queried object
+- `UserId` in session metadata
+
+### Rule 6: Customer names, email addresses, and contact data
+
+Any field that may contain customer-facing PII — even if the field name does
+not explicitly indicate PII — must be treated as sensitive:
+
+- Email addresses: `<email_redacted>`
+- Phone numbers: `<phone_redacted>`
+- Customer names appearing in intent summaries: `<name_redacted>`
+- Mailing or billing addresses: `<address_redacted>`
+- Social Security Numbers or national ID numbers: `<national_id_redacted>`
+
+If a field's content is ambiguous, redact it and note the redaction in the
+audit envelope.
+
+### Rule 7: Instance URLs and API endpoints
+
+Do not include Salesforce instance URLs (e.g., `https://myorg.my.salesforce.com`),
+Data Cloud endpoints, or any URL that reveals org-specific infrastructure.
+Reference only the org alias in the audit envelope.
+
+### Rule 8: Agent version suffix on DeveloperName
+
+The `_vN` version suffix on `DeveloperName` (e.g., `OrderServiceAgent_v9`)
+reveals internal versioning cadence. Omit the suffix from all output; use the
+base API name only.
+
+### Rule 9: Encrypted fields (Shield PE / PMLE)
+
+If a query inadvertently touches a field marked `encrypted: true` in the
+sObject describe response, skip the field entirely. Do not emit the field
+name or any placeholder that implies a value was retrieved. List skipped
+fields in the audit envelope under `redactions_applied`.
+
+---
+
+## Regulated Vertical Handling
+
+When the target org is identified as serving a regulated vertical — Health Cloud,
+Financial Services Cloud, Government Cloud, or any other edition that may host
+protected data — apply the following additional controls:
+
+1. **Flag the org** in the audit envelope: `regulated_vertical_flag: true`.
+2. **Apply all standard redaction rules** with zero exceptions.
+3. **Do not share output externally** without routing through
+   `salesforce-compliance-privacy-agent` first.
+4. **If any anomaly is detected**, the downstream recommendation must include
+   `salesforce-compliance-privacy-agent` regardless of anomaly severity.
+5. **Retain no session metadata** beyond the current conversation turn. Do not
+   write session IDs, participant IDs, or any session-linked identifier to files
+   or intermediate storage.
+
+**Regulated vertical detection signals:**
+- Org alias includes terms: `hc`, `health`, `fsc`, `financial`, `gov`, `govt`
+- `sf org display` output includes industry-specific features
+- User states the org is used for Health Cloud, Financial Services Cloud,
+  Government Cloud, or equivalent
+- Org metadata includes custom objects with obvious regulated-data names
+  (e.g., `Patient__c`, `MedicalRecord__c`, `LoanApplication__c`)
+
+If there is any doubt, treat the org as regulated.
+
+---
+
+## Audit Envelope Redaction Record
+
+Every redaction applied must be recorded in the audit envelope under
+`redactions_applied`. Each entry includes:
+
+```yaml
+redactions_applied:
+  - type: "session_content"
+    reason: "aggregate-only policy — session text content never emitted"
+  - type: "user_id"
+    reason: "participant IDs replaced with placeholder per redaction rule 5"
+  - type: "record_id"
+    reason: "record IDs hashed for dedup tracking; raw IDs not emitted"
+  - type: "pii"
+    field: "<FieldApiName>"
+    reason: "field may contain customer contact data"
+```
+
+The audit envelope redaction record allows downstream reviewers to understand
+what was removed and why, without the redacted content being present.
+
+---
+
+## Human-in-the-Loop Path for Session Content Access
+
+If a human operator or security reviewer has a legitimate need to access raw
+session content for debugging a specific production issue (e.g., investigating
+a confirmed hallucination in a specific session):
+
+1. The requester must initiate the request through `salesforce-live-guard-agent`.
+2. `salesforce-live-guard-agent` routes for explicit human approval from an
+   authorized data steward.
+3. The approved requester accesses session content directly via Salesforce
+   Setup or the STDM API with their own authenticated credentials — not
+   through this skill.
+4. The access event must be logged in the org's Event Monitoring stream
+   .
+5. This skill is not involved in the content access — it stops at the
+   referral to `salesforce-live-guard-agent`.
+
+**This path exists for genuine debugging needs only.** It must not be used
+as a routine observability mechanism. Production session content access
+requires human authorization every time.
+
+---
+
+## What This Skill Is Permitted to Emit
+
+Summary of what IS and IS NOT emitted:
+
+| Data type | Permitted | Redaction applied |
+|---|---|---|
+| Session count (integer) | Yes | None |
+| Avg faithfulness score (float) | Yes | None |
+| Avg answer relevance score (float) | Yes | None |
+| Avg quality score (float, 1-5) | Yes | None |
+| Quality distribution (counts per score) | Yes | None |
+| Action invocation count (integer) | Yes | None |
+| Action error count (integer) | Yes | None |
+| Abandonment rate (float) | Yes | None |
+| End type counts (USER_ENDED, AGENT_ENDED, UNKNOWN) | Yes | None |
+| Top intents (intent label + count) | Yes | Intent label may be paraphrased by LLM — treat as potentially sensitive; emit without modification |
+| Subagent API names (for score breakdown) | Yes | Omit version suffix |
+| Session IDs | No | Replaced with `<session_id_placeholder>` |
+| User messages | No | Never fetched |
+| Agent responses | No | Never fetched |
+| LLM prompts | No | Never fetched |
+| LLM responses | No | Never fetched |
+| Request/response summaries (LLM paraphrases) | No | Never emitted |
+| Quality reasoning text | No | Never emitted |
+| Record IDs (raw) | No | Replaced with placeholder or hash |
+| User IDs / participant IDs | No | Replaced with placeholder |
+| Org IDs | No | Replaced with placeholder |
+| OAuth tokens | No | Never fetched or emitted |
+| Instance URLs | No | Omitted; org alias used instead |