jumpstart-mode 1.1.13 → 1.1.14

package/.jumpstart/skills/multi-cloud-architecture/SKILL.md

@@ -0,0 +1,181 @@
+---
+name: multi-cloud-architecture
+description: Design multi-cloud architectures using a decision framework to select and integrate services across AWS, Azure, and GCP. Use when building multi-cloud systems, avoiding vendor lock-in, or leveraging best-of-breed services from multiple providers.
+---
+
+# Multi-Cloud Architecture
+
+Decision framework and patterns for architecting applications across AWS, Azure, and GCP.
+
+## Purpose
+
+Design cloud-agnostic architectures and make informed decisions about service selection across cloud providers.
+
+## When to Use
+
+- Design multi-cloud strategies
+- Migrate between cloud providers
+- Select cloud services for specific workloads
+- Implement cloud-agnostic architectures
+- Optimize costs across providers
+
+## Cloud Service Comparison
+
+### Compute Services
+
+| AWS     | Azure               | GCP             | Use Case           |
+| ------- | ------------------- | --------------- | ------------------ |
+| EC2     | Virtual Machines    | Compute Engine  | IaaS VMs           |
+| ECS     | Container Instances | Cloud Run       | Containers         |
+| EKS     | AKS                 | GKE             | Kubernetes         |
+| Lambda  | Functions           | Cloud Functions | Serverless         |
+| Fargate | Container Apps      | Cloud Run       | Managed containers |
+
+### Storage Services
+
+| AWS     | Azure           | GCP             | Use Case       |
+| ------- | --------------- | --------------- | -------------- |
+| S3      | Blob Storage    | Cloud Storage   | Object storage |
+| EBS     | Managed Disks   | Persistent Disk | Block storage  |
+| EFS     | Azure Files     | Filestore       | File storage   |
+| Glacier | Archive Storage | Archive Storage | Cold storage   |
+
+### Database Services
+
+| AWS         | Azure            | GCP           | Use Case        |
+| ----------- | ---------------- | ------------- | --------------- |
+| RDS         | SQL Database     | Cloud SQL     | Managed SQL     |
+| DynamoDB    | Cosmos DB        | Firestore     | NoSQL           |
+| Aurora      | PostgreSQL/MySQL | Cloud Spanner | Distributed SQL |
+| ElastiCache | Cache for Redis  | Memorystore   | Caching         |
+
+**Reference:** See `references/service-comparison.md` for complete comparison
+
+## Multi-Cloud Patterns
+
+### Pattern 1: Single Provider with DR
+
+- Primary workload in one cloud
+- Disaster recovery in another
+- Database replication across clouds
+- Automated failover
+
+### Pattern 2: Best-of-Breed
+
+- Use best service from each provider
+- AI/ML on GCP
+- Enterprise apps on Azure
+- General compute on AWS
+
+### Pattern 3: Geographic Distribution
+
+- Serve users from nearest cloud region
+- Data sovereignty compliance
+- Global load balancing
+- Regional failover
+
+### Pattern 4: Cloud-Agnostic Abstraction
+
+- Kubernetes for compute
+- PostgreSQL for database
+- S3-compatible storage (MinIO)
+- Open source tools
+
+## Cloud-Agnostic Architecture
+
+### Use Cloud-Native Alternatives
+
+- **Compute:** Kubernetes (EKS/AKS/GKE)
+- **Database:** PostgreSQL/MySQL (RDS/SQL Database/Cloud SQL)
+- **Message Queue:** Apache Kafka (MSK/Event Hubs/Confluent)
+- **Cache:** Redis (ElastiCache/Azure Cache/Memorystore)
+- **Object Storage:** S3-compatible API
+- **Monitoring:** Prometheus/Grafana
+- **Service Mesh:** Istio/Linkerd
+
+### Abstraction Layers
+
+```
+Application Layer
+    ↓
+Infrastructure Abstraction (Terraform)
+    ↓
+Cloud Provider APIs
+    ↓
+AWS / Azure / GCP
+```
+
+## Cost Comparison
+
+### Compute Pricing Factors
+
+- **AWS:** On-demand, Reserved, Spot, Savings Plans
+- **Azure:** Pay-as-you-go, Reserved, Spot
+- **GCP:** On-demand, Committed use, Preemptible
+
+### Cost Optimization Strategies
+
+1. Use reserved/committed capacity (30-70% savings)
+2. Leverage spot/preemptible instances
+3. Right-size resources
+4. Use serverless for variable workloads
+5. Optimize data transfer costs
+6. Implement lifecycle policies
+7. Use cost allocation tags
+8. Monitor with cloud cost tools
+
+**Reference:** See `references/multi-cloud-patterns.md`
+
+## Migration Strategy
+
+### Phase 1: Assessment
+
+- Inventory current infrastructure
+- Identify dependencies
+- Assess cloud compatibility
+- Estimate costs
+
+### Phase 2: Pilot
+
+- Select pilot workload
+- Implement in target cloud
+- Test thoroughly
+- Document learnings
+
+### Phase 3: Migration
+
+- Migrate workloads incrementally
+- Maintain dual-run period
+- Monitor performance
+- Validate functionality
+
+### Phase 4: Optimization
+
+- Right-size resources
+- Implement cloud-native services
+- Optimize costs
+- Enhance security
+
+## Best Practices
+
+1. **Use infrastructure as code** (Terraform/OpenTofu)
+2. **Implement CI/CD pipelines** for deployments
+3. **Design for failure** across clouds
+4. **Use managed services** when possible
+5. **Implement comprehensive monitoring**
+6. **Automate cost optimization**
+7. **Follow security best practices**
+8. **Document cloud-specific configurations**
+9. **Test disaster recovery** procedures
+10. **Train teams** on multiple clouds
+
+## Reference Files
+
+- `references/service-comparison.md` - Complete service comparison
+- `references/multi-cloud-patterns.md` - Architecture patterns
+
+## Related Skills
+
+- `terraform-module-library` - For IaC implementation
+- `cost-optimization` - For cost management
+- `hybrid-cloud-networking` - For connectivity

package/.jumpstart/skills/non-functional-requirements-elicitor/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: non-functional-requirements-elicitor
+description: Force AI-specific non-functional requirements into scope before they get discovered late, covering latency, throughput, accuracy and recall targets, groundedness, hallucination tolerance, cost ceilings per call and per workflow, availability, fallback behavior, observability, evaluation gates, security, privacy, residency, and human-in-the-loop expectations. Use this whenever the user needs NFRs, SLAs, SLOs, quality targets, cost guardrails, fallback design, eval thresholds, or production readiness criteria for an AI build, even when phrased as "what does good look like," "how fast does it need to be," "what if the model is wrong," or "what should we measure."
+argument-hint: Paste the AI use case, user audience, criticality, known constraints (regulatory, cost, latency), and any existing SLAs or quality expectations.
+---
+
+# Non-Functional Requirements Elicitor
+
+## Positioning
+
+Use this skill to surface the AI-specific non-functional requirements that traditional NFR templates miss — accuracy targets, hallucination tolerance, groundedness, eval gates, fallback behavior, per-call cost ceilings, human-in-the-loop expectations — before they bite the Pod late. The output should be a defensible NFR set that can drive architecture, evaluation, and acceptance criteria.
+
+## When To Use This
+
+- The user is scoping an AI use case and has not yet defined targets for latency, accuracy, cost, fallback, or eval gates.
+- The team needs an NFR set that complements functional scope before SOW, PRD, or production readiness review.
+- Phrased as "what does good look like," "what's the SLA," "how do we know it's working," or "what if it fails."
+
+## When Not To Use This Alone
+
+- Do not use this for measuring business outcomes. Use `../success-metrics-designer/SKILL.md` for KPIs and instrumentation.
+- Do not use this for governance posture. Use `../agent-governance/SKILL.md`.
+- Do not use this for data quality scoring. Use `../data-readiness-assessor/SKILL.md`.
+
+## Required Inputs
+
+- AI use case and primary user audience,
+- criticality (advisory, decision-support, autonomous action),
+- regulatory or contractual constraints,
+- volume estimates (calls / day, peak concurrency),
+- known cost or budget ceilings,
+- prior SLAs or quality expectations on adjacent systems,
+- tolerance for being wrong (and what wrong looks like).
+
+## NFR Categories
+
+Cover all of these explicitly. If the answer is "no requirement," capture that as a deliberate decision, not an omission.
+
+1. Performance — p50, p95, p99 latency; throughput; concurrency.
+2. Quality — task success rate, accuracy, recall/precision where measurable, groundedness, citation behavior, hallucination tolerance.
+3. Cost — per-call, per-user, per-workflow ceilings; caching, routing, batching expectations.
+4. Availability — uptime target, planned maintenance, regional failover.
+5. Fallback — what happens on model failure, tool failure, low confidence, rate-limit hit.
+6. Human-in-the-loop — when human review is required, who, with what SLA.
+7. Evaluation gates — pre-deploy and continuous eval thresholds; release block conditions.
+8. Observability — tracing, prompt and tool logs, eval logs, cost metrics, drift signals.
+9. Security and access — authn / authz model, secret handling, tool permissioning.
+10. Privacy and data handling — minimization, retention, redaction, training-data exclusion, regional residency.
+11. Compliance — regulatory regimes that apply (HIPAA, GDPR, EU AI Act, SR 11-7, FedRAMP).
+12. Adoption — accessibility, language, change management hooks.
+
+See [references/nfr-catalog.md](./references/nfr-catalog.md) for elicitation prompts and anchor examples per category.
+
+## Workflow
+
+1. Restate the use case and criticality class. Criticality drives default targets.
+2. Walk each NFR category with the elicitation prompts. Capture target, rationale, source, and confidence.
+3. For every target, define how it will be measured and where the signal lives (logs, evals, traces).
+4. Surface conflicts (e.g., low latency vs. retrieval depth, cost ceiling vs. accuracy target) and flag them as architectural tradeoffs.
+5. Mark any unverified target as `Assumed — needs sponsor confirmation`.
+6. Produce the NFR register using [templates/nfr-register-template.md](./templates/nfr-register-template.md).
+
+## Output Mode
+
+Default artifact:
+
+- Executive summary (criticality class, top three NFR risks, top three open decisions).
+- Criticality classification and rationale.
+- NFR register table (category, requirement, target, measurement, source, confidence).
+- Conflict register (tradeoffs the sponsor must adjudicate).
+- Acceptance criteria implications (which NFRs become release-blocking).
+- Open questions for sponsor or architect.
+
+## Working Rules
+
+1. No target without a measurement plan.
+2. No measurement plan without a signal source.
+3. Default to release-blocking for safety, privacy, and groundedness on regulated or high-impact use cases.
+4. Do not invent specific numbers. Offer anchor ranges and require sponsor confirmation.

package/.jumpstart/skills/non-functional-requirements-elicitor/evals/evals.json

@@ -0,0 +1,31 @@
+{
+  "skill_name": "non-functional-requirements-elicitor",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "We are scoping an AI assistant for advisor research at a wealth management firm. The sponsor said 'make it fast and accurate' and that's all we have. Pull the NFRs out before SOW. Volume is roughly 600 advisors, 5 queries each per day. Regulated under SEC and they care about audit.",
+      "expected_output": "A complete NFR register covering performance, quality, cost, availability, fallback, HITL, eval gates, observability, security, privacy, compliance, and adoption — with anchor targets, measurement plans, and confidence labels.",
+      "files": [],
+      "assertions": [
+        {"text": "NFR register covers all 12 categories named in the catalog"},
+        {"text": "Each requirement has a measurement plan with a signal source"},
+        {"text": "Targets are presented as anchors requiring sponsor confirmation, not invented as hard numbers"},
+        {"text": "Compliance section explicitly addresses SEC / SR 11-7-style audit obligations"},
+        {"text": "Output includes a conflict register surfacing tradeoffs (e.g., latency vs. retrieval depth)"},
+        {"text": "Release-blocking NFRs are clearly identified"}
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "This is an autonomous agent that processes refunds up to $500 in a retail back-office system. We need NFRs that match the criticality. What changes vs. an advisory copilot?",
+      "expected_output": "Higher criticality classification with HITL on irreversible actions, release-blocking eval gates, stricter fallback design, and audit trail requirements.",
+      "files": [],
+      "assertions": [
+        {"text": "Criticality is classified as autonomous-action and the rationale is explicit"},
+        {"text": "HITL is required for actions above a defined threshold or for irreversible actions"},
+        {"text": "Eval gates are release-blocking"},
+        {"text": "Audit trail is captured per consequential decision"}
+      ]
+    }
+  ]
+}

package/.jumpstart/skills/non-functional-requirements-elicitor/references/nfr-catalog.md

@@ -0,0 +1,97 @@
+# NFR Catalog for AI Builds
+
+Use this as an elicitation aid. For each category, pose the probing questions, capture the target, and identify how it will be measured.
+
+## 1. Performance
+
+- What p50 / p95 / p99 latency does the user experience tolerate? Compare to current process baseline.
+- What concurrency at peak (calls/min)? What growth profile in 12 months?
+- Are there long-running operations (>5s)? If so, what is the UX (streaming, progress, async result)?
+
+## 2. Quality
+
+- What is the task definition? What counts as "right"?
+- Is there a labelable golden set? Who labels it?
+- Targets: task success rate, faithfulness / groundedness, citation accuracy, recall, precision, exact-match where appropriate.
+- Hallucination tolerance: zero, bounded, advisory-only?
+- How does the user recognize a wrong answer? Can they?
+
+## 3. Cost
+
+- Per-call ceiling (tokens × price + tool calls).
+- Per-user / per-day envelope.
+- Per-workflow envelope.
+- Caching, routing (small vs. large model), batching expectations.
+- Cost alarming thresholds.
+
+## 4. Availability
+
+- Uptime target (99.0 / 99.5 / 99.9). Anchor to underlying model and platform availability.
+- Maintenance windows; regional failover; multi-region read.
+- Degraded modes (read-only, cached-only, manual fallback).
+
+## 5. Fallback
+
+- What happens on model failure, tool failure, low confidence, rate-limit hit, content filter trip?
+- Does the system retry, escalate to human, return cached, or fail closed?
+- How is the user told?
+
+## 6. Human-in-the-loop
+
+- Which actions require human approval before execution?
+- SLA for human review.
+- Who is the human; when are they on; what is their UX?
+- What is logged for audit?
+
+## 7. Evaluation gates
+
+- Pre-deploy eval thresholds and which assertions are release-blocking.
+- Continuous eval cadence and traffic sample.
+- Eval set ownership and refresh cadence.
+- Adversarial / red-team gates.
+
+## 8. Observability
+
+- Tracing for prompts, retrievals, tool calls, model calls.
+- Eval logs with version pinning of prompt, model, and tool spec.
+- Cost and token metrics per request.
+- Drift signals (input distribution shift, output quality drop, refusal rate, latency creep).
+- PII handling in logs.
+
+## 9. Security and access
+
+- Authn / authz model for users.
+- Identity for the agent itself; tool permissioning; least privilege.
+- Secret handling and rotation.
+- Threat model coverage (prompt injection, data exfil, tool abuse, jailbreak).
+
+## 10. Privacy and data handling
+
+- Data minimization in prompts and logs.
+- Retention by data type and pipeline stage (prompts, retrievals, eval data, fine-tune data).
+- Redaction or masking before logging.
+- Training-data exclusion clauses for the chosen model.
+- Regional residency.
+
+## 11. Compliance
+
+- HIPAA, GDPR, CCPA, GLBA, PCI-DSS, SR 11-7, EU AI Act, FedRAMP, sectoral codes.
+- Audit and explainability obligations.
+- Disclosure to end users that AI is in use.
+
+## 12. Adoption
+
+- Accessibility (screen reader, keyboard, color contrast).
+- Language and locale.
+- Onboarding and in-product help.
+- Escalation path when the user disagrees with the AI.
+
+## Default targets by criticality
+
+| Criticality | Quality | Latency p95 | HITL | Eval cadence |
+|---|---|---|---|---|
+| Advisory copilot | Best-effort with citations | <3s | Optional | Weekly |
+| Decision-support | High groundedness, citation required | <3s | Recommended on key actions | Daily |
+| Autonomous action | Release-blocking eval gates | <5s | Required for irreversible actions | Per release + continuous |
+
+These are anchors, not commitments. Sponsor confirms per use case.

package/.jumpstart/skills/non-functional-requirements-elicitor/templates/nfr-register-template.md

@@ -0,0 +1,51 @@
+# NFR Register: <Use Case>
+
+**Owner:** <name>
+**Date:** <YYYY-MM-DD>
+**Criticality class:** Advisory / Decision-support / Autonomous action
+**Confidence legend:** Confirmed / Assumed / Unknown
+
+## Executive summary
+
+- Criticality and rationale.
+- Top three NFR risks.
+- Top three open decisions for sponsor.
+
+## NFR register
+
+| Category | Requirement | Target | Measurement | Signal source | Release-blocking? | Confidence | Source |
+|---|---|---|---|---|---|---|---|
+| Performance — latency p95 | | <ms> | request span | tracing | Y/N | | |
+| Performance — throughput | | <calls/min> | counter | telemetry | | | |
+| Quality — task success | | <%> | golden eval set | eval log | Y/N | | |
+| Quality — groundedness | | <%> | judge eval | eval log | Y/N | | |
+| Quality — citation rate | | <%> | output parser | eval log | | | |
+| Cost — per call | | $<x> | tokens × price + tools | telemetry | | | |
+| Cost — per workflow | | $<x> | aggregated | telemetry | | | |
+| Availability | | <%> | uptime monitor | platform | | | |
+| Fallback — model failure | | behavior | runbook | runbook | | | |
+| HITL — actions requiring approval | | list | UX flag | audit log | Y/N | | |
+| Eval gate — release | | thresholds | eval suite | CI | Y/N | | |
+| Observability | | what is logged | trace + eval | tracing | | | |
+| Security | | controls | threat model | sec review | | | |
+| Privacy | | minimization, retention, residency | DPIA | privacy review | | | |
+| Compliance | | regimes | mapped controls | legal | | | |
+| Adoption | | accessibility, locale, escalation | UX standards | UX review | | | |
+
+## Conflict register (tradeoffs)
+
+| Conflict | Option A | Option B | Recommendation | Sponsor decision required by |
+|---|---|---|---|---|
+| Latency p95 vs. retrieval depth | | | | |
+| Cost ceiling vs. model size | | | | |
+| Autonomy vs. HITL | | | | |
+
+## Acceptance criteria implications
+
+- Release-blocking NFRs: <list, each tied to evidence>
+- Pre-launch closures: <list, owner, date>
+- Fast-followers (post-launch): <list>
+
+## Open questions
+
+- <question> — <owner> — <by date>

package/.jumpstart/skills/observability-design-for-ai/SKILL.md

@@ -0,0 +1,93 @@
+---
+name: observability-design-for-ai
+description: 'Design tracing, eval logging, drift detection, cost telemetry, and red-team signal for an AI system so the team can debug, govern, and improve in production. Use this whenever the user is shaping how to monitor an AI build, even when phrased as "how do we know it works in production", "set up logging for the agent", "track drift", "monitor costs", or "instrument the copilot".'
+argument-hint: 'Provide the use case, platform, expected volume, governance/audit requirements, and any existing observability stack.'
+---
+
+# Observability Design For AI
+
+## Positioning
+
+AI systems fail differently than traditional services. They are slower, costlier, and degrade silently. Use this skill to design the observability that makes those failures visible — traces that show the full prompt-and-tool path, eval logs tied to each call, drift detection on the metrics that matter, cost telemetry at multiple grains, and red-team signal feeding back into the eval set.
+
+A good design lets the team:
+
+1. Reproduce any production output (prompt, model, retrieved context, tool calls).
+2. Spot quality drift before users do.
+3. Spot cost and latency anomalies before finance does.
+4. Feed real production failures back into the eval set without manual stitching.
+
+## When To Use This
+
+- Designing observability for a new AI build heading to pilot or production.
+- An existing system is having "it works on my machine" or silent quality drift.
+- A regulator, auditor, or sponsor asks for evidence of monitoring.
+- Cost or latency anomalies are appearing without a way to triage them.
+
+## When Not To Use This
+
+- Do not use this in place of [`evaluation-harness-designer`](../evaluation-harness-designer/SKILL.md). The eval suite is the offline test; observability is the production signal. Pair them.
+- Do not use this in place of [`agent-governance`](../agent-governance/SKILL.md) for the policy framework.
+- Do not use this for non-AI service observability; classic APM patterns suffice.
+
+## The Five Telemetry Streams
+
+1. **Trace** — every call: prompt hash, model, retrieved context refs, tool calls, latency per step, outcome, cost.
+2. **Eval log** — judge scores attached to a sampled subset of production calls; drives drift detection.
+3. **Cost** — tokens × price + tool/storage costs, attributed by user, tenant, feature, slice.
+4. **Drift / quality signals** — retrieval recall on a held-out set, calibration, refusal rate, edit rate, override rate.
+5. **Safety / red-team signal** — injection attempts detected, jailbreak rate, PII filter hits, abuse patterns.
+
+See [references/telemetry-catalog.md](./references/telemetry-catalog.md) for the canonical fields per stream.
+
+## Workflow
+
+1. Inventory call types (assistant turn, tool call, retrieval, agent step). Each gets a span schema.
+2. Define the trace span schema using [templates/trace-schema.md](./templates/trace-schema.md). Include redaction rules.
+3. Define the eval-on-production sampling plan and attach scores to traces.
+4. Define cost attribution: by user, tenant, feature flag, model, prompt version.
+5. Define drift signals and alert thresholds.
+6. Define safety telemetry and the path back into eval and prompt updates.
+7. Pick the platform: Application Insights + Log Analytics, OpenTelemetry + LangSmith / Langfuse / Arize / Phoenix, Foundry, Datadog, etc. Match client stack.
+8. Wire dashboards: Pod-internal triage, sponsor-facing trust, finance-facing cost.
+9. Capture in [templates/observability-spec.md](./templates/observability-spec.md).
+
+## Output Structure
+
+1. Use case and call types.
+2. Trace span schema.
+3. Eval-on-production plan.
+4. Cost attribution model.
+5. Drift signals and thresholds.
+6. Safety telemetry.
+7. Platform and storage with retention.
+8. Dashboards and alerts.
+9. Redaction and access control on logs.
+10. Risks, assumptions, next validation.
+
+A worked output is in [examples/example-grounded-assistant-observability.md](./examples/example-grounded-assistant-observability.md).
+
+## Common Pitfalls
+
+- **Logging the prompt without the response.** Or vice versa. Both, with the prompt hash, or you cannot reproduce.
+- **PII in logs.** Redact at the boundary; never depend on downstream cleanup.
+- **One trace stream for "everything".** Tool calls, retrieval, and generation need shared correlation but distinct spans.
+- **No eval-on-production.** Offline eval drifts; production sampling is what catches it.
+- **Cost only at the model call.** Tool, storage, and retrieval costs hide; aggregate them.
+- **Too noisy alerts.** Alerts that fire constantly are ignored. Tune thresholds with baselines.
+- **No path from incident to eval.** A production failure that does not become a new eval case is a failure that will repeat.
+
+## Quality Bar
+
+- Every call has a reproducible trace (prompt, context refs, tool calls, outcome, cost).
+- Production sampling is scored and feeds drift detection.
+- Costs are attributable to user, tenant, feature, model, and prompt version.
+- Logs are redacted, access-controlled, and retention-bounded.
+
+## Related Skills
+
+- `evaluation-harness-designer` — offline eval; pairs with production sampling.
+- `agent-governance` — policy on what is logged and who can access.
+- `red-team-playbook-builder` — feeds detected injection cases into eval.
+- `cost-anomaly-detector` (planned) — uses cost telemetry from this design.
+- `failure-mode-and-effects-ai` — failure modes drive what to instrument.

package/.jumpstart/skills/observability-design-for-ai/evals/evals.json

@@ -0,0 +1,32 @@
+{
+  "skill_name": "observability-design-for-ai",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Design observability for our new internal AI assistant going to pilot. Azure-only stack, 3 tenants, ~500 users, expected 20k calls/day. Sponsor wants weekly trust evidence; finance wants tenant-level cost; security wants injection telemetry.",
+      "expected_output": "An observability spec covering traces, eval-on-production, cost attribution, drift, safety telemetry, dashboards, and access control.",
+      "expectations": [
+        "Defines a reproducible trace span schema covering LLM, retrieval, tool, and memory calls with redaction.",
+        "Includes eval-on-production sampling with judge model, metrics, and drift alert thresholds.",
+        "Attributes cost by tenant, user, feature, prompt version, and model.",
+        "Calls out drift signals (retrieval probe, refusal/edit rate, calibration) with thresholds.",
+        "Designs safety telemetry (injection detector, PII filter, new-pattern feedback into eval set).",
+        "Defines dashboards for ops triage, sponsor trust, and finance cost separately.",
+        "Specifies retention, redaction at ingest, and tiered access control with audit on raw content."
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Production AI copilot has had two silent quality regressions in three weeks. We have logs of every prompt and response. What's missing in our observability and how do we fix it fast?",
+      "expected_output": "A gap analysis and a remediation plan that adds eval-on-production, drift detection, and feedback loops.",
+      "expectations": [
+        "Identifies missing eval-on-production sampling as the primary gap behind silent regressions.",
+        "Recommends drift signals (retrieval probe, refusal rate, edit/override rate) and alert thresholds.",
+        "Recommends tying prompt_version and model_version to every trace so regressions can be bisected.",
+        "Adds calibration cadence for any LLM-judge in the loop.",
+        "Adds a feedback loop from production failures into the eval set.",
+        "Sequences fixes by impact and effort, getting drift alerts live within days."
+      ]
+    }
+  ]
+}

package/.jumpstart/skills/observability-design-for-ai/examples/example-grounded-assistant-observability.md

@@ -0,0 +1,53 @@
+# Example: Grounded Customer Assistant Observability
+
+## Call types
+- LLM generation, retrieval, tool call (CRM lookup, ticket create), memory read/write.
+
+## Trace platform
+- App Insights + OTel collector forwarding to Langfuse for AI-specific dashboards.
+- correlation_id from the front-end request; passes through to all child spans.
+
+## Eval-on-production
+- 5% of all calls sampled; 100% of refusal-required and low-confidence cases.
+- Judge: GPT-4o, calibrated weekly against 50-case human set.
+- Metrics: faithfulness, refusal correctness, citation correctness, brand safety.
+- Drift alert: > 3% drop on any metric vs. release baseline (rolling 7-day).
+
+## Cost attribution
+- Aggregated daily by tenant, feature (search vs. ticket-create), prompt_version, model.
+- Burn alert at 80% of monthly tenant budget.
+
+## Drift signals
+- Daily retrieval probe (50 frozen queries) measuring recall@k.
+- Refusal rate by topic; alert on > 2σ change.
+- Edit rate from HITL approval queue; alert weekly.
+
+## Safety telemetry
+- Inline prompt-injection detector on retrieved content; logs hit + chunk_id.
+- PII regex + classifier on inputs and outputs; high-severity hits block response.
+- New injection patterns auto-queued for eval-set review.
+
+## Storage and retention
+| Stream | Store | Retention | Redaction |
+|---|---|---|---|
+| Traces (redacted) | App Insights | 30d | yes |
+| Traces (raw, restricted) | ADLS | 7d | no, restricted access |
+| Eval scores | Langfuse | 90d | yes |
+| Cost rollups | SQL | 13 months | n/a |
+| Safety hits | App Insights + queue | 90d | yes |
+
+## Dashboards and alerts
+- Triage: top errors, p95 latency, refusal rate, injection hits.
+- Trust: weekly faithfulness, refusal correctness trend.
+- Cost: daily burn vs. budget, top tenants, prompt-version cost diff.
+- On-call: alerts go to Pod tech lead PagerDuty; safety alerts also go to security.
+
+## Access control
+- Ops: redacted traces only.
+- SRE with PII training: joined view.
+- Security: raw store via audited break-glass.
+
+## Risks, assumptions, next validation
+- Risk: judge drift; mitigated by weekly calibration.
+- Assumption: Langfuse approved by client InfoSec.
+- Next: dry-run alert thresholds for two weeks and tune.