perf-skills 1.0.0

package/perf-skills/references/topics/observability.md

@@ -0,0 +1,201 @@
+# Observability in Performance Testing
+
+Observability is what bridges "the test shows slow responses" to "the database is saturating its connection pool at 300 VUs." Without correlated server-side metrics, performance test analysis is guesswork.
+
+---
+
+## The Three Pillars
+
+| Pillar | What It Tells You | Tools |
+|---|---|---|
+| **Metrics** | Numeric measurements over time (CPU, RPS, latency histograms) | Prometheus, InfluxDB, Datadog, CloudWatch |
+| **Logs** | Event records with context (error details, request IDs) | Elasticsearch, Loki, Splunk, CloudWatch Logs |
+| **Traces** | End-to-end request path across services | Jaeger, Tempo, Datadog APM, Dynatrace |
+
+A complete performance investigation uses all three, correlated by time and request ID.
+
+---
+
+## Minimal Observability Stack for Performance Testing
+
+### Open Source (self-hosted)
+```
+Load Tool (k6/JMeter)
+    │ pushes metrics
+    ▼
+InfluxDB  ←──────────────────────────────────
+    │                                         │
+    ▼                                    Prometheus
+Grafana Dashboard                            │
+    │                              Node Exporter (host metrics)
+    ▼                              cAdvisor (container metrics)
+Alerting (Slack/PagerDuty)         JMX Exporter (JVM metrics)
+```
+
+### Cloud-native
+```
+Load Tool → Grafana Cloud (k6 native) → Grafana Dashboard
+App → Datadog / Dynatrace / New Relic → correlated in single pane
+```
+
+---
+
+## Correlating Load Test with APM
+
+The most powerful workflow: overlay load metrics with APM metrics on the same timeline.
+
+**Step 1:** Add a test identifier to all load-tool requests:
+```javascript
+// k6: tag all requests
+export const options = {
+  tags: { testRun: __ENV.BUILD_ID || 'manual', env: 'staging' }
+};
+```
+```bash
+# JMeter: set custom header on all requests
+# In HTTP Header Manager (at Test Plan level):
+X-Load-Test-Run: ${__P(build.id,manual)}
+X-Load-Test-Phase: ${__P(phase,load)}
+```
+
+**Step 2:** In your APM, filter traces and metrics by the test tag — isolate test traffic from organic traffic.
+
+**Step 3:** Align timelines — zoom in on the window when p95 degraded and look at:
+- Service CPU and memory during that window
+- DB query duration spike
+- Thread pool queue depth
+- Cache hit rate drop
+
+---
+
+## Prometheus + Grafana Setup
+
+### k6 → Prometheus (remote write)
+```bash
+k6 run --out experimental-prometheus-rw \
+  -e K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write \
+  -e K6_PROMETHEUS_RW_TREND_STATS='p(50),p(95),p(99)' \
+  script.js
+```
+
+### JMeter → InfluxDB
+1. Install **Backend Listener** config element.
+2. Set implementation: `InfluxdbBackendListenerClient`
+3. Config: `influxdbUrl=http://influx:8086`, `measurement=jmeter`, `token=<token>`
+
+### Useful Grafana Dashboards
+| Dashboard | Grafana ID | For |
+|---|---|---|
+| k6 Load Testing Results | 2587 | k6 + InfluxDB |
+| JMeter Load Test | 1152 | JMeter + InfluxDB |
+| Node Exporter Full | 1860 | Host system metrics |
+| JVM Overview | 4701 | Java app JVM metrics |
+| Kubernetes Cluster | 7249 | K8s pod metrics |
+
+---
+
+## JVM Metrics (Java Applications)
+
+For Java services under test, always collect JVM metrics alongside load tool metrics.
+
+### Expose via JMX Exporter (Prometheus)
+```yaml
+# jmx_config.yaml
+rules:
+  - pattern: java.lang<type=Memory><HeapMemoryUsage>used
+    name: jvm_heap_used_bytes
+  - pattern: java.lang<type=GarbageCollector, name=(.+)><CollectionTime>
+    name: jvm_gc_collection_seconds_total
+    labels:
+      gc: $1
+```
+
+### Key JVM Metrics to Monitor
+| Metric | Normal | Warning |
+|---|---|---|
+| Heap used % | < 70% | > 85% = GC pressure |
+| GC pause duration (p99) | < 200ms | > 500ms = impacting response time |
+| GC frequency | Occasional | Continuous = memory leak |
+| Thread pool queue size | 0–10 | Growing = throughput ceiling reached |
+| JDBC pool active connections | < 80% | > 90% = connection starvation |
+
+---
+
+## Log Correlation
+
+During a performance test, log volume explodes. Use structured logging and filtering:
+
+### Mark test traffic in logs
+```java
+// Spring Boot: use MDC to add test context
+MDC.put("testRun", request.getHeader("X-Load-Test-Run"));
+// Log4j/Logback will include this in every log line for this thread
+```
+
+### Useful log queries during test (Elasticsearch/Kibana)
+```
+# Error spike investigation
+testRun: "build-42" AND level: ERROR
+| group by logger, message
+| order by count desc
+
+# Slow transactions
+testRun: "build-42" AND duration_ms: >500
+| group by endpoint
+| percentile(duration_ms, 95)
+```
+
+---
+
+## Distributed Tracing
+
+Traces show the end-to-end path of a single request across microservices — invaluable for pinpointing *which service* in a chain is slow.
+
+### Setup for load testing
+- Confirm tracing is active in the target environment.
+- After the test, sample traces from the slowest requests (p99 window).
+- Use the trace waterfall to find which service/DB call accounts for the most latency.
+
+### Adding trace context to load tool requests
+```javascript
+// k6: pass trace context (Jaeger B3 format)
+const traceId = randomString(32, '0123456789abcdef');
+http.get(url, {
+    headers: {
+        'X-B3-TraceId': traceId,
+        'X-B3-SpanId': randomString(16, '0123456789abcdef'),
+        'X-B3-Sampled': '1',
+        'X-Load-Test-Run': __ENV.BUILD_ID
+    }
+});
+```
+
+---
+
+## Synthetic Monitoring (Post-Test)
+
+After a load test validates a build, set up **synthetic monitors** to continuously test a minimal user journey in production:
+
+| Tool | Description |
+|---|---|
+| Grafana Synthetic Monitoring | k6 scripts run on schedule from cloud probes |
+| Datadog Synthetics | Browser + API tests from global locations |
+| AWS CloudWatch Synthetics (Canaries) | Node.js scripts on schedule |
+| Checkly | API + Browser checks with k6 integration |
+
+Synthetic monitors catch regressions that slip through staging without requiring another full load test.
+
+---
+
+## Performance Testing Observability Checklist
+
+- [ ] APM tool active and capturing traces in test environment
+- [ ] Server metrics (CPU, memory, disk I/O) dashboards ready before test start
+- [ ] JVM metrics exposed (for Java services)
+- [ ] DB slow query logging enabled and queryable
+- [ ] Load tool pushing real-time metrics to dashboard (InfluxDB, Grafana Cloud, etc.)
+- [ ] Test run tagged with build ID / run ID for filtering
+- [ ] Alert thresholds set (so team is notified if something breaks during test)
+- [ ] Log aggregation active and searchable
+- [ ] Baseline metrics screenshot taken before test starts (for comparison)
+- [ ] Metrics retention configured to keep results for trend analysis

package/perf-skills/references/topics/production-testing.md

@@ -0,0 +1,214 @@
+# Production & Staging Performance Testing
+
+Running tests in production is not inherently dangerous — it's a discipline. Many organizations run production performance testing routinely. The key is controlled blast radius, observability, and rollback readiness.
+
+---
+
+## Staging Environment Testing
+
+### What Makes Staging Valid for Performance Testing
+
+Staging is only a reliable proxy for production if:
+
+| Factor | Requirement |
+|---|---|
+| **Hardware** | Same instance type, CPU, memory as production |
+| **Architecture** | Same number of replicas, same DB tier, same cache config |
+| **Data volume** | DB has similar row counts to production (schema-level) |
+| **Network** | Similar topology (no shortcutting the load balancer) |
+| **Dependencies** | Real downstream services (not mocks for performance tests) |
+
+Using mocks for load tests is only appropriate for component-level isolation tests — full user journey tests must hit real (or realistic stub) dependencies.
+
+### Staging Limitations
+- Cache hit rates will differ if data set is too small.
+- Third-party integrations may behave differently (rate limits are usually lower in test).
+- Infrastructure auto-scaling may not be configured identically.
+- Cold-start performance distorts early-stage results — pre-warm before measuring.
+
+### Pre-Warm Strategy
+```bash
+# Run a smoke test (1 VU) for 5 minutes before starting the load test
+# This warms:
+# - JVM JIT compilation
+# - Connection pools
+# - Application-level caches
+# - CDN edge caches (if applicable)
+
+k6 run --vus 1 --duration 5m --tag phase=warmup warmup.js
+k6 run --vus 500 --duration 20m --tag phase=load main.js
+```
+
+---
+
+## Production Testing Strategies
+
+### Strategy 1: Canary Deployment + Load Test
+
+Route a small % of production traffic to the new release and observe:
+
+```
+Production Traffic
+    │
+    ├─ 95% → Stable instances (v1.0)
+    └─ 5%  → Canary instances (v1.1)  ← Monitor closely
+```
+
+Gradually increase canary traffic percentage while monitoring:
+- Error rate (should not increase)
+- p95 response time (should not increase)
+- Business metrics (conversion rate, etc.)
+
+**Tools:** AWS ALB weighted target groups, Kubernetes Argo Rollouts, Istio traffic splitting, NGINX split_clients.
+
+### Strategy 2: Shadow Testing (Traffic Mirroring)
+
+Mirror production traffic to a shadow cluster — it receives all production requests but its responses are discarded:
+
+```
+Real User Request
+    │
+    ├─ Live → Production (v1) → Response to user
+    └─ Mirror → Shadow (v2)  → Response discarded (no user impact)
+```
+
+Shadow cluster processes real production workload — perfect for validating performance of a new version without any user impact.
+
+**Tools:** AWS ALB request mirroring, Istio `mirror`, NGINX `mirror` directive.
+
+```yaml
+# Istio traffic mirroring
+apiVersion: networking.istio.io/v1alpha3
+kind: VirtualService
+spec:
+  http:
+  - route:
+    - destination:
+        host: my-service-v1
+    mirror:
+      host: my-service-v2
+    mirrorPercentage:
+      value: 100.0
+```
+
+### Strategy 3: Synthetic Load Injection in Production
+
+Inject artificial load at low volumes (typically 5–10% of peak) during off-peak hours:
+
+- Use a dedicated load generator with production-class credentials.
+- Tag synthetic requests to exclude from business metrics dashboards.
+- Use a synthetic user pool (not real users).
+- Monitor production SLAs during injection.
+
+**When to use:** Validate autoscaling behavior, warm new nodes before peak, run regression checks post-deployment.
+
+### Strategy 4: Chaos Engineering (Resilience Testing)
+
+Intentionally inject failures to test how the system degrades:
+
+| Experiment | Tool | Tests |
+|---|---|---|
+| Kill a pod/instance | Chaos Monkey, LitmusChaos | Failover speed, error rate |
+| Throttle CPU/memory | Stress-ng, LitmusChaos | Degradation under resource pressure |
+| Introduce network latency | tc netem, Chaos Mesh | Timeout handling, retry logic |
+| Kill the database | LitmusChaos | Circuit breaker, fallback |
+| Saturate disk I/O | fio, LitmusChaos | Log rotation, disk-full handling |
+
+```bash
+# Inject 100ms network latency + 10% packet loss to a pod
+kubectl exec -it chaos-pod -- \
+  tc qdisc add dev eth0 root netem delay 100ms loss 10%
+```
+
+**Always combine chaos with a baseline load** — run at 50% of expected peak so the system has load to respond to when faults occur.
+
+---
+
+## Pre-Production Performance Gates
+
+### Gate Criteria (must pass before production deploy)
+
+```
+1. Smoke test (1 VU):           Zero errors, correct responses
+2. Load test (target VUs):      p95 < SLA, error rate < 1%
+3. Stress test (1.5× target):   Graceful degradation, no data corruption
+4. Regression comparison:       p95 within 10% of previous passing run
+```
+
+### Exemptions and Escalation
+- If a test fails, categorize: **known regression** (documented, fix tracked) vs **unexpected regression** (block deploy).
+- SLA breach of < 5% may be acceptable with product owner sign-off.
+- Emergency hotfixes may skip stress test but must pass smoke + load.
+
+---
+
+## Safety Controls for Production Testing
+
+### Mandatory Safety Controls
+
+| Control | Implementation |
+|---|---|
+| **Blast radius limit** | Never inject more than 10–20% of prod capacity without approval |
+| **Kill switch** | A single command / button to stop all injectors immediately |
+| **Auto-abort on error spike** | Test stops automatically if error rate exceeds threshold |
+| **Rollback plan** | Documented and tested rollback procedure ready |
+| **Communication** | Ops/SRE on standby; incident channel open |
+| **Synthetic user tagging** | All test requests tagged to exclude from real user metrics |
+| **No real user data** | Synthetic test data only — never process real PII in load tests |
+
+### Auto-abort configuration
+
+```javascript
+// k6: auto-abort on error spike
+export const options = {
+  thresholds: {
+    http_req_failed: [{
+      threshold: 'rate<0.05',  // Abort if error rate > 5%
+      abortOnFail: true,
+      delayAbortEval: '30s',   // Wait 30s before aborting (avoid false positives)
+    }],
+  },
+};
+```
+
+```bash
+# JMeter: stop test on error rate threshold via Backend Listener + custom alerting
+# Or use Taurus wrapper:
+bzt test.jmx \
+  -o modules.passfail.checks[0].subject=fail \
+  -o modules.passfail.checks[0].threshold=5% \
+  -o modules.passfail.checks[0].condition=over \
+  -o modules.passfail.checks[0].timeframe=60s \
+  -o modules.passfail.checks[0].stop=true
+```
+
+---
+
+## Environments Progression
+
+```
+Developer Laptop (smoke, 1 VU)
+    ↓
+CI Environment (smoke + light load, 10–25 VUs, fast feedback)
+    ↓
+Staging / Performance Environment (full load + stress tests)
+    ↓
+Pre-Production / Mirror (shadow tests, canary validation)
+    ↓
+Production (synthetic monitoring, canary, controlled injection)
+```
+
+Each environment adds fidelity; the production stage assumes all previous stages passed.
+
+---
+
+## Incident Response During a Performance Test
+
+If something breaks during a test:
+
+1. **Immediately stop the test** (kill switch — stop all injectors).
+2. **Capture current state** — snapshot metrics, take thread dumps (Java), capture logs.
+3. **Assess impact** — are real users affected? Is the failure isolated to test traffic?
+4. **Roll back if needed** — restore previous version, restart services.
+5. **Preserve evidence** — don't restart services without capturing logs and heap dumps.
+6. **Post-mortem** — document what broke, at what load, what the root cause was.

package/perf-skills/references/topics/protocol-testing.md

@@ -0,0 +1,274 @@
+# Protocol-Specific Performance Testing
+
+Covers gRPC, GraphQL, WebSocket/SSE, and message queue (Kafka, RabbitMQ, SQS) load testing — protocols with unique challenges beyond standard HTTP/REST.
+
+---
+
+## gRPC Performance Testing
+
+gRPC uses HTTP/2, Protocol Buffers, and persistent connections — fundamentally different from REST at the wire level.
+
+### Key Challenges
+
+| Challenge | Why It Matters |
+|---|---|
+| **Protobuf compilation** | Tests require compiled `.proto` definitions; can't just send raw JSON |
+| **Connection multiplexing** | HTTP/2 multiplexes streams over a single connection — fewer connections needed but different bottleneck profile |
+| **Streaming** | Unary, server-streaming, client-streaming, and bidirectional-streaming each need different test strategies |
+| **Metadata vs headers** | gRPC metadata is the equivalent of HTTP headers; auth tokens go here |
+| **Error codes** | gRPC uses its own status codes (OK, UNAVAILABLE, DEADLINE_EXCEEDED) — not HTTP status codes |
+
+### Tool Support
+
+| Tool | gRPC Support | Notes |
+|---|---|---|
+| **k6** | `k6/grpc` built-in module | Supports unary and streaming; load `.proto` or use reflection |
+| **Gatling** | `gatling-grpc` plugin | Scala/Java DSL; supports unary and streaming |
+| **JMeter** | `gRPC Sampler` plugin | GUI-based; requires proto descriptor file |
+| **ghz** | Dedicated gRPC benchmarking tool | CLI-only; excellent for quick benchmarks |
+
+### k6 gRPC Example
+
+```javascript
+import grpc from 'k6/grpc';
+import { check, sleep } from 'k6';
+
+const client = new grpc.Client();
+client.load(['definitions'], 'hello.proto');
+
+export default function () {
+  client.connect('grpc-server:50051', { plaintext: true });
+
+  const response = client.invoke('hello.HelloService/SayHello', {
+    greeting: 'perf-test',
+  });
+
+  check(response, {
+    'status is OK': (r) => r && r.status === grpc.StatusOK,
+    'has message': (r) => r && r.message.reply !== '',
+  });
+
+  client.close();
+  sleep(1);
+}
+```
+
+### gRPC Testing Considerations
+
+- **Connection reuse**: Keep connections open across iterations; don't connect/close per request.
+- **Streaming throughput**: For server-streaming RPCs, measure messages-per-second, not just request latency.
+- **Deadline propagation**: Set gRPC deadlines in tests to match production timeouts.
+- **Load balancer awareness**: gRPC over HTTP/2 with persistent connections can cause uneven load across backends — test with client-side load balancing or L7 proxy.
+- **Protobuf payload size**: Binary encoding is smaller than JSON — adjust throughput expectations accordingly.
+
+---
+
+## GraphQL Performance Testing
+
+GraphQL introduces query complexity as a variable — the same endpoint can serve trivially cheap or devastatingly expensive requests.
+
+### Key Challenges
+
+| Challenge | Why It Matters |
+|---|---|
+| **Variable query cost** | A single endpoint can serve queries with wildly different server costs |
+| **N+1 resolver problem** | Nested queries can trigger cascading DB calls; load tests expose this at scale |
+| **Query depth / complexity** | Deep nesting can cause exponential resolver execution |
+| **Batched queries** | Clients may send multiple operations in one request |
+| **Persisted queries** | Production may use query allowlists; ad-hoc queries get rejected |
+
+### Testing Strategy
+
+1. **Catalog production queries**: Extract real queries from logs or APM traces — don't invent synthetic queries.
+2. **Categorize by cost**: Light (single field), medium (nested 2 levels), heavy (deep joins, lists).
+3. **Build a realistic query mix**: Weight tests by actual production query distribution.
+4. **Test with and without caching**: GraphQL caching (DataLoader, CDN) dramatically changes performance profiles.
+
+### k6 GraphQL Example
+
+```javascript
+import http from 'k6/http';
+import { check, sleep } from 'k6';
+
+const GRAPHQL_ENDPOINT = `${__ENV.BASE_URL}/graphql`;
+
+const QUERIES = {
+  lightQuery: `query { user(id: "${__VU}") { name email } }`,
+  heavyQuery: `query { users(first: 100) { edges { node { name orders(last: 10) { totalAmount items { name } } } } } }`,
+};
+
+export default function () {
+  // 80% light queries, 20% heavy queries
+  const query = Math.random() < 0.8 ? QUERIES.lightQuery : QUERIES.heavyQuery;
+
+  const res = http.post(GRAPHQL_ENDPOINT, JSON.stringify({ query }), {
+    headers: {
+      'Content-Type': 'application/json',
+      'Authorization': `Bearer ${__ENV.TOKEN}`,
+    },
+    tags: { name: 'POST /graphql' },
+  });
+
+  check(res, {
+    'status 200': (r) => r.status === 200,
+    'no errors': (r) => !r.json('errors'),
+    'has data': (r) => r.json('data') !== null,
+  });
+
+  sleep(1);
+}
+```
+
+### GraphQL-Specific Metrics to Track
+
+- **Query complexity score** (if server exposes it)
+- **Resolver execution time** (via APM / tracing)
+- **DataLoader batch efficiency** (cache hit rate)
+- **Error rate by query type** (complexity-related timeouts vs auth errors)
+
+---
+
+## WebSocket / SSE Performance Testing
+
+Persistent connections have fundamentally different scaling characteristics from request/response HTTP.
+
+### Key Challenges
+
+| Challenge | Why It Matters |
+|---|---|
+| **Connection count** | Each VU holds an open connection — tests memory/fd limits, not just CPU |
+| **Message throughput** | Measure messages/sec independently from connection count |
+| **Backpressure** | What happens when the server can't send fast enough? |
+| **Reconnection behavior** | Dropped connections should auto-reconnect; test the reconnect storm |
+| **Fan-out cost** | Broadcasting to N connections has O(N) server cost |
+
+### Tool Support
+
+| Tool | WebSocket | SSE |
+|---|---|---|
+| **k6** | `k6/ws` module | Not built-in; use HTTP streaming |
+| **Gatling** | Full WebSocket DSL | Not built-in |
+| **JMeter** | WebSocket Sampler plugin | Not built-in |
+| **Artillery** | Built-in `ws` engine | Built-in SSE |
+
+### k6 WebSocket Example
+
+```javascript
+import ws from 'k6/ws';
+import { check, sleep } from 'k6';
+
+export default function () {
+  const url = `${__ENV.WS_URL}/ws/chat`;
+  const params = { headers: { 'Authorization': `Bearer ${__ENV.TOKEN}` } };
+
+  const res = ws.connect(url, params, function (socket) {
+    socket.on('open', () => {
+      socket.send(JSON.stringify({ type: 'subscribe', channel: 'updates' }));
+    });
+
+    socket.on('message', (msg) => {
+      const data = JSON.parse(msg);
+      check(data, {
+        'has type': (d) => d.type !== undefined,
+      });
+    });
+
+    socket.on('error', (e) => {
+      console.error('WS error:', e.error());
+    });
+
+    // Hold connection open for 30 seconds, sending periodic pings
+    socket.setInterval(() => {
+      socket.send(JSON.stringify({ type: 'ping' }));
+    }, 5000);
+
+    socket.setTimeout(() => {
+      socket.close();
+    }, 30000);
+  });
+
+  check(res, { 'WS status 101': (r) => r && r.status === 101 });
+  sleep(1);
+}
+```
+
+### WebSocket Testing Strategy
+
+1. **Connection ramp**: Gradually open connections — don't blast 10k connections at once.
+2. **Separate connection test from message test**: First, find max stable connections; then test message throughput at a stable connection count.
+3. **Measure server-side**: File descriptor count, memory per connection, event loop lag.
+4. **Test reconnection storms**: Kill server, observe client reconnection behavior and server recovery.
+
+---
+
+## Message Queue / Event Streaming Testing
+
+Testing Kafka, RabbitMQ, SQS, and similar systems requires measuring producer throughput, consumer lag, and end-to-end latency.
+
+### Key Challenges
+
+| Challenge | Why It Matters |
+|---|---|
+| **Producer throughput** | Messages/sec the system can ingest |
+| **Consumer lag** | How far behind consumers fall under load |
+| **End-to-end latency** | Time from produce to consume — the real user-facing metric |
+| **Partition scaling** | Kafka throughput scales with partitions; test with realistic partition counts |
+| **Message ordering** | Under load, verify ordering guarantees still hold |
+| **Dead letter queues** | Verify failed messages route correctly under pressure |
+
+### Tool Support
+
+| Tool | Kafka | RabbitMQ | SQS |
+|---|---|---|---|
+| **k6 (xk6-kafka)** | Yes | No (use HTTP management API) | No |
+| **JMeter** | Kafka plugin / JMS Sampler | JMS Sampler | AWS SDK Sampler |
+| **kafka-producer-perf-test** | Built-in Kafka tool | N/A | N/A |
+| **rabbitmq-perf-test** | N/A | Official RabbitMQ tool | N/A |
+
+### k6 Kafka Example (xk6-kafka)
+
+```javascript
+import { Writer, Reader, Connection } from 'k6/x/kafka';
+
+const writer = new Writer({ brokers: ['kafka:9092'], topic: 'perf-test' });
+const reader = new Reader({ brokers: ['kafka:9092'], topic: 'perf-test', groupID: 'perf-group' });
+
+export default function () {
+  // Produce
+  writer.produce({
+    messages: [{
+      key: `key-${__VU}-${__ITER}`,
+      value: JSON.stringify({ orderId: `${__VU}-${__ITER}`, timestamp: Date.now() }),
+    }],
+  });
+
+  // Consume (in a separate scenario ideally)
+  const messages = reader.consume({ limit: 1 });
+  // Validate message content
+}
+
+export function teardown() {
+  writer.close();
+  reader.close();
+}
+```
+
+### Message Queue Testing Strategy
+
+1. **Test producers and consumers separately first**, then together.
+2. **Measure consumer lag over time** — a growing lag under steady load indicates a bottleneck.
+3. **Test with realistic message sizes** — a 100-byte message vs a 1MB payload have very different throughput ceilings.
+4. **Verify idempotency** — under load with retries, duplicate messages should not corrupt state.
+5. **Test partition rebalancing** — add/remove consumers during a test to verify rebalance behavior.
+
+---
+
+## Protocol Testing Checklist
+
+- [ ] Proto definitions / schemas compiled and available to test tool
+- [ ] Connection reuse strategy defined (persistent vs per-request)
+- [ ] Streaming scenarios identified and scripted separately from unary/request-response
+- [ ] Message throughput targets defined (messages/sec, not just RPS)
+- [ ] Backpressure / flow control behavior validated
+- [ ] Error codes and retry behavior tested (gRPC status codes, AMQP nacks, etc.)
+- [ ] End-to-end latency measured (not just request latency)