@assertrx/dino 1.0.10 → 1.0.12

package/skills/promptfoo/SKILL.md

@@ -0,0 +1,121 @@
+---
+name: promptfoo
+description: Use when writing, running, or debugging promptfoo AI evaluations — promptfooconfig.yaml structure, providers (OpenAI/Anthropic/HTTP/echo), test cases with vars, assert types (contains, is-json, javascript, similar, llm-rubric), CLI usage (eval, view, JSON output), thresholds, and CI integration. Use when the user asks about LLM evals, prompt evaluation, model output testing, or setting up a promptfoo project.
+---
+
+# Promptfoo — LLM / Prompt Evaluations
+
+Author, run, and interpret [promptfoo](https://www.promptfoo.dev/) evaluations. Promptfoo evaluates LLM prompts, agents, and model-backed endpoints against test cases with deterministic and model-graded assertions, producing a pass/fail result like a test runner.
+
+## When to Use Promptfoo
+
+| Goal | Approach |
+|------|----------|
+| Verify prompt/model changes don't regress outputs | Eval suite with deterministic asserts on known inputs |
+| Compare prompts or models side by side | Multiple `prompts` or `providers` in one config — promptfoo renders a matrix |
+| Test subjective quality (tone, faithfulness) | `llm-rubric` assert (author it with the **write-judge-prompt** skill) |
+| Test an HTTP endpoint that wraps a model | `http` provider hitting the real endpoint |
+| Sanity-check the harness itself, no API needed | `echo` provider — instant, offline, zero cost |
+| Find vulnerable/unsafe outputs | `npx promptfoo redteam` (separate command, own config) |
+
+## Project Layout
+
+```
+promptfoo/
+  promptfooconfig.yaml   # the eval config: prompts + providers + tests
+  prompts/
+    system.txt           # prompt templates ({{var}} placeholders) — or inline
+  tests/                 # optional: split test cases into YAML/CSV/JSON files
+  README.md              # how to run, env vars needed
+```
+
+A single `promptfooconfig.yaml` is enough for a skeleton project; split prompts/tests into files only when the config grows.
+
+## Config Anatomy (promptfooconfig.yaml)
+
+```yaml
+description: "Customer support agent — smoke evals"
+
+# What to evaluate. One or more prompt templates; {{vars}} come from tests.
+prompts:
+  - file://promptfoo/prompts/system.txt
+  # or inline:
+  # - "Translate the following to French: {{text}}"
+
+# Who answers. Multiple providers = comparison matrix.
+providers:
+  - openai:gpt-4o-mini          # needs OPENAI_API_KEY
+  - anthropic:claude-3-5-haiku-latest  # needs ANTHROPIC_API_KEY
+  # Local/zero-cost provider for harness checks:
+  # - echo
+
+# Test cases: inputs + expected behavior.
+tests:
+  - description: "Greeting is friendly"
+    vars:
+      message: "hi there"
+    assert:
+      - type: contains
+        value: "hello"
+      - type: llm-rubric
+        value: "The reply is polite and offers help"
+
+# Assertions applied to every test (defaults; per-test asserts extend these).
+defaultTest:
+  assert:
+    - type: latency
+      threshold: 5000   # ms
+```
+
+Full assert reference: [reference/assert-types.md](reference/assert-types.md). Provider reference: [reference/providers.md](reference/providers.md).
+
+## Running Evals
+
+```bash
+# Run the default config (promptfooconfig.yaml in cwd)
+npx promptfoo@latest eval
+
+# Run a specific config, fail the command on failed asserts (default)
+npx promptfoo@latest eval -c promptfoo/promptfooconfig.yaml
+
+# Machine-readable results (for CI or parsing)
+npx promptfoo@latest eval -c promptfoo/promptfooconfig.yaml --output json
+
+# Interactive web viewer of the last results
+npx promptfoo@latest view
+```
+
+- **Exit code**: `0` when all asserts pass, non-zero when any test fails or the config is invalid — safe for CI.
+- **Cost control**: `--repeat 1` (default), keep test counts small, prefer cheap models while iterating.
+- **Concurrency**: defaults are conservative; raise via `--max-concurrency N` for larger suites.
+
+## Test-Design Rules
+
+1. **Deterministic first.** Use `contains`, `is-json`, `javascript`, and `similar` before model-graded asserts — they are reproducible, free, and debuggable.
+2. **Rubric only for the subjective.** For `llm-rubric`, write the judge criteria with the **write-judge-prompt** skill, and calibrate it with **validate-evaluator** before trusting results.
+3. **Small, sharp suites.** 5–20 targeted cases beat 200 vague ones. Generate bulk synthetic variants with **generate-synthetic-data** only when you need coverage/robustness checks.
+4. **No secrets in configs.** API keys come from env (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, …) — never commit them.
+5. **Thresholds over vibes.** Set `defaultTest` latency/cost guards so regressions fail loudly even when content asserts pass.
+
+## CI Integration (GitHub Actions)
+
+```yaml
+- name: Run prompt evals
+  env:
+    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+  run: npx promptfoo@latest eval -c promptfoo/promptfooconfig.yaml --output json > results.json
+```
+
+## Interpreting Results
+
+- Each row = one test × (prompt × provider). Red cells list the exact assert that failed with actual vs expected — read that first.
+- For `llm-rubric` failures, inspect the judge's reason in the viewer before assuming the model is wrong; a mis-calibrated judge is a common failure mode (see **eval-audit** skill).
+- Flaky content asserts (`contains` on chatty models) → relax to `contains-any`, `similar` with a threshold, or a better-scoped rubric.
+
+## Related Skills in This Package
+
+- **write-judge-prompt** — author the `llm-rubric` criteria.
+- **validate-evaluator** — calibrate judges against human labels before trusting scores.
+- **eval-audit** — audit an existing eval suite for missing error analysis, vanity metrics, etc.
+- **generate-synthetic-data** — bulk-generate test inputs across behavioral dimensions.
+- **error-analysis** — categorize failures after a run to decide what to fix first.

package/skills/promptfoo/reference/assert-types.md

@@ -0,0 +1,74 @@
+# Promptfoo Assert Types Reference
+
+Assertions live under each test's `assert:` list (and `defaultTest.assert:` for suite-wide defaults). Every assert evaluates to pass/fail against the provider output. Prefer deterministic types; use model-graded types only for subjective criteria.
+
+## Deterministic (reproducible, free)
+
+| Type | What it checks | Key fields |
+|------|----------------|-----------|
+| `contains` | Output includes a substring | `value` |
+| `not-contains` | Output must not include a substring | `value` |
+| `contains-any` | Output includes at least one of several substrings | `value: [..]` |
+| `contains-all` | Output includes all of several substrings | `value: [..]` |
+| `regex` | Output matches a pattern | `value` (regex string) |
+| `equals` | Exact match (trim/normalize first) | `value` |
+| `is-json` | Output parses as JSON; optionally validates a JSON schema | `value` (JSON schema) |
+| `javascript` | Custom predicate over output/context | `value` (JS expression returning boolean/number; `output`, `context` available) |
+| `python` | Same, in Python | `value` (code) |
+| `similar` | Embedding similarity to expected text | `value`, `threshold` (0–1, default 0.75) |
+| `latency` | Response time guard | `threshold` (ms) |
+| `cost` | Per-call cost guard | `threshold` (USD) |
+| `contains-sql` / `is-valid-sql` / `is-valid-function-call` / `is-valid-openai-function-call` / `is-valid-url` / `no-refusal` / `perplexity` / `perplexity-score` | Domain-specific structural checks | varies |
+
+## Model-graded (subjective criteria — judge LLM involved)
+
+| Type | What it checks | Notes |
+|------|----------------|-------|
+| `llm-rubric` | Output satisfies natural-language criteria | Judge prompt in `value`; author it with the **write-judge-prompt** skill. Optional `rubricPrompt` to override the default judge prompt template |
+| `factuality` | Output is factually consistent with a reference | `value` = ground truth, `subject` = original input |
+| `answer-relevance` | Output addresses the input | no extra fields |
+| `context-recall` / `context-relevance` / `context-faithfulness` | RAG quality: retrieved context vs question/answer | see **evaluate-rag** skill |
+| `classifier` | Label the output with a model-chosen category | `value` = list of allowed labels |
+
+## Grading Model
+
+Model-graded asserts run on a grader LLM. Configure it suite-wide:
+
+```yaml
+# Use a different model for judging than for the system under test
+evaluateOptions:
+  # or per-assert via the assert's own provider field
+providers: []
+```
+
+Simplest per-assert override:
+
+```yaml
+- type: llm-rubric
+  value: "Answer states the refund window and cites the policy doc."
+  # provider: anthropic:claude-3-5-haiku-latest   # optional judge override
+```
+
+## Weighted & Threshold Asserts
+
+```yaml
+tests:
+  - description: "Balanced checks"
+    vars: { q: "what is your return policy" }
+    assert:
+      - type: contains
+        value: "30 days"
+        weight: 2          # counts double in the score
+      - type: llm-rubric
+        value: "Polite and on-topic"
+        weight: 1
+    threshold: 0.8         # test passes only if score >= 0.8
+```
+
+## Suite-wide Pass Threshold
+
+```yaml
+# Fail the whole eval if less than 90% of tests pass
+tests:
+  - threshold: 0.9
+```

package/skills/promptfoo/reference/providers.md

@@ -0,0 +1,70 @@
+# Promptfoo Providers Reference
+
+Providers answer prompts. List one or more under `providers:` — multiple entries render a comparison matrix across all tests.
+
+## Model Providers
+
+```yaml
+providers:
+  - openai:gpt-4o-mini                 # needs OPENAI_API_KEY
+  - anthropic:claude-3-5-haiku-latest  # needs ANTHROPIC_API_KEY
+  - bedrock:us.anthropic.claude-3-5-sonnet-20241022-v2:0
+  - azureopenai:gpt-4o:my-deployment   # needs AZURE_OPENAI_API_BASE etc.
+  - google:gemini-1.5-flash            # needs GOOGLE_API_KEY
+  - ollama:chat:llama3.1               # local, no key
+```
+
+Custom model params (temperature, max tokens) use the long form:
+
+```yaml
+providers:
+  - id: anthropic:claude-3-5-haiku-latest
+    config:
+      temperature: 0
+      max_tokens: 512
+```
+
+## HTTP Provider (test any endpoint that wraps an LLM)
+
+```yaml
+providers:
+  - id: http
+    config:
+      url: https://api.example.com/v1/answer
+      method: POST
+      headers:
+        Content-Type: application/json
+        Authorization: Bearer {{ENV.API_TOKEN}}
+      body:
+        model: support-agent
+        messages: [{ role: user, content: "{{prompt}}" }]
+      transformResponse: json.response   # JSONPath-ish pick of the text to assert on
+```
+
+`{{prompt}}` is the rendered prompt template; `{{ENV.NAME}` reads environment variables — no secrets in the config file.
+
+## Echo Provider (offline harness checks)
+
+Returns the rendered prompt verbatim — no network, no cost. Use it to validate the config, wiring, and deterministic asserts before spending tokens (note: plain `- echo`, not `- echo:text` — the colon shorthand was removed in recent versions):
+
+```yaml
+providers:
+  - echo
+```
+
+## File/Script Providers
+
+- `file://path/to/provider.py` or `.js` — full control (custom agent, chain, RAG pipeline); export a `CallApi(prompt, context)` function.
+- Python file providers let you wrap internal SDKs not reachable over HTTP.
+
+## Environment Variables
+
+| Provider | Variable(s) |
+|----------|-------------|
+| OpenAI | `OPENAI_API_KEY` |
+| Anthropic | `ANTHROPIC_API_KEY` |
+| Google | `GOOGLE_API_KEY` / `GEMINI_API_KEY` |
+| Azure OpenAI | `AZURE_OPENAI_API_BASE`, `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_DEPLOYMENT_NAME` |
+| HTTP | anything referenced via `{{ENV.*}}` |
+
+Load from `.env` files or the shell; never commit keys.

package/skills/testing-resilience/SKILL.md

@@ -0,0 +1,230 @@
+---
+name: testing-resilience
+description: Use when creating chaos engineering or resilience tests with k6 and xk6-disruptor. Use when the user wants to inject faults (pod failures, network delays, HTTP errors) into Kubernetes services during load tests, validate circuit breakers, or test system recovery behavior.
+---
+
+# Testing Resilience with k6
+
+Create chaos and resilience tests using xk6-disruptor to inject faults into Kubernetes services during k6 load tests. Validates system behavior under failure conditions.
+
+## Overview
+
+xk6-disruptor injects faults into Kubernetes pods and services while k6 generates load, allowing you to verify:
+- System survives pod failures
+- Graceful degradation under network issues
+- Circuit breaker behavior
+- Recovery time after fault removal
+- Auto-scaling response to failures
+
+## Quick Start
+
+```javascript
+import http from 'k6/http';
+import { check, sleep } from 'k6';
+import { ServiceDisruptor } from 'k6/x/disruptor';
+
+export const options = {
+  scenarios: {
+    load: {
+      executor: 'constant-arrival-rate',
+      rate: 100,
+      timeUnit: '1s',
+      duration: '2m',
+      preAllocatedVUs: 50,
+    },
+  },
+  thresholds: {
+    http_req_failed: ['rate<0.3'],        // Allow up to 30% errors during chaos
+    http_req_duration: ['p(95)<3000'],     // Relaxed threshold during chaos
+  },
+};
+
+export function setup() {
+  // Inject fault: Add 500ms delay to 50% of requests
+  const disruptor = new ServiceDisruptor('my-service', 'default');
+  const fault = {
+    averageDelay: '500ms',
+    errorRate: 0.1,           // 10% of requests return errors
+    errorCode: 503,
+  };
+  disruptor.injectHTTPFaults(fault, '60s');
+}
+
+export default function () {
+  const res = http.get('http://my-service.default.svc.cluster.local:8080/api');
+  check(res, {
+    'not server error': (r) => r.status < 500 || r.status === 503,
+  });
+  sleep(0.1); // Intentional: adds backpressure per VU to avoid overwhelming the disrupted service
+}
+```
+
+## Prerequisites
+
+- **Kubernetes cluster** with target services deployed
+- **k6 with xk6-disruptor** extension (or use automatic resolution)
+- **RBAC permissions** for the k6 pod to manage target pods
+
+## Disruptor Types
+
+### ServiceDisruptor
+
+Targets all pods behind a Kubernetes Service:
+
+```javascript
+import { ServiceDisruptor } from 'k6/x/disruptor';
+
+const disruptor = new ServiceDisruptor('service-name', 'namespace');
+```
+
+### PodDisruptor
+
+Targets specific pods by label selector:
+
+```javascript
+import { PodDisruptor } from 'k6/x/disruptor';
+
+const disruptor = new PodDisruptor({
+  namespace: 'default',
+  select: { labels: { app: 'my-app' } },
+});
+```
+
+## Fault Types
+
+### HTTP Faults
+
+Inject delays and errors into HTTP responses:
+
+```javascript
+disruptor.injectHTTPFaults(
+  {
+    averageDelay: '500ms',      // Add delay to responses
+    delayVariation: '100ms',    // Delay jitter
+    errorRate: 0.1,             // 10% error rate
+    errorCode: 503,             // Error HTTP status code
+    port: 8080,                 // Target port
+    exclude: '/health',         // Exclude health check endpoint
+  },
+  '60s'                         // Fault duration
+);
+```
+
+### gRPC Faults
+
+Inject faults into gRPC services:
+
+```javascript
+disruptor.injectGrpcFaults(
+  {
+    averageDelay: '500ms',
+    errorRate: 0.2,
+    statusCode: 14,             // UNAVAILABLE
+    statusMessage: 'service overloaded',
+  },
+  '60s'
+);
+```
+
+### Pod Termination
+
+Kill pods to test recovery:
+
+```javascript
+disruptor.terminatePods({
+  count: 1,                     // Number of pods to terminate
+  interval: '30s',              // Wait between terminations
+});
+```
+
+See [reference/fault-injection.md](reference/fault-injection.md) for detailed fault configuration.
+
+## Resilience Test Patterns
+
+### Pattern 1: Graceful Degradation
+
+Verify system degrades gracefully under partial failure:
+
+```javascript
+export function setup() {
+  const disruptor = new ServiceDisruptor('backend-service', 'default');
+  disruptor.injectHTTPFaults({
+    averageDelay: '1000ms',
+    errorRate: 0.2,
+    errorCode: 503,
+  }, '90s');
+}
+```
+
+**Success criteria:** System returns degraded responses instead of cascading failures.
+
+### Pattern 2: Circuit Breaker Validation
+
+Verify circuit breaker activates under error conditions:
+
+```javascript
+export function setup() {
+  const disruptor = new ServiceDisruptor('downstream-service', 'default');
+  disruptor.injectHTTPFaults({
+    errorRate: 0.8,             // 80% errors to trigger circuit breaker
+    errorCode: 500,
+  }, '60s');
+}
+```
+
+**Success criteria:** Upstream service returns fallback responses quickly (circuit open).
+
+### Pattern 3: Recovery Test
+
+Verify system recovers after fault is removed:
+
+```javascript
+export const options = {
+  scenarios: {
+    load: {
+      executor: 'constant-arrival-rate',
+      rate: 50,
+      timeUnit: '1s',
+      duration: '5m',           // Total test: 5 minutes
+      preAllocatedVUs: 30,
+    },
+  },
+};
+
+export function setup() {
+  const disruptor = new ServiceDisruptor('my-service', 'default');
+  // Inject fault for only 2 minutes (test runs 5 minutes)
+  disruptor.injectHTTPFaults({
+    averageDelay: '2000ms',
+    errorRate: 0.5,
+    errorCode: 503,
+  }, '120s');  // Fault active for 2m, then 3m of recovery
+}
+```
+
+**Success criteria:** Metrics return to baseline within acceptable time after fault removal.
+
+See [reference/resilience-patterns.md](reference/resilience-patterns.md) for more patterns.
+
+## Thresholds for Resilience Tests
+
+Resilience tests need relaxed thresholds compared to normal load tests. Use tag-based thresholds to set different criteria for chaos and recovery phases:
+
+```javascript
+export const options = {
+  thresholds: {
+    // During chaos: accept higher error rates
+    http_req_failed: ['rate<0.5'],                           // Up to 50% errors acceptable
+    // Overall response time upper bound (includes chaos phase)
+    http_req_duration: ['p(95)<5000'],                       // 5s max even during chaos
+    // Strict threshold for recovery phase only (tagged requests)
+    'http_req_duration{phase:recovery}': ['p(95)<500'],      // After recovery, back to normal
+  },
+};
+```
+
+## Related Skills
+
+- For generating load test scripts: `/k6:generating-api-load-tests`
+- For CI/CD and Kubernetes setup: `/k6:operating-k6-in-ci-cd`
+- For scenario design: `/k6:designing-test-scenarios`

package/skills/testing-resilience/reference/fault-injection.md

@@ -0,0 +1,153 @@
+# Fault Injection Reference
+
+## ServiceDisruptor
+
+Targets all pods behind a Kubernetes Service.
+
+```javascript
+import { ServiceDisruptor } from 'k6/x/disruptor';
+
+const disruptor = new ServiceDisruptor('service-name', 'namespace');
+```
+
+### injectHTTPFaults(fault, duration)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `averageDelay` | string | Average delay added to responses (e.g., `'500ms'`) |
+| `delayVariation` | string | Variation around averageDelay (e.g., `'100ms'`) |
+| `errorRate` | float | Fraction of requests returning errors (0.0-1.0) |
+| `errorCode` | integer | HTTP status code for injected errors |
+| `errorBody` | string | Response body for injected errors |
+| `port` | integer | Target port (required if service has multiple ports) |
+| `exclude` | string | URL path prefix to exclude from faults |
+
+```javascript
+disruptor.injectHTTPFaults({
+  averageDelay: '500ms',
+  delayVariation: '100ms',
+  errorRate: 0.1,
+  errorCode: 503,
+  errorBody: '{"error": "service unavailable"}',
+  port: 8080,
+  exclude: '/health',
+}, '60s');
+```
+
+### injectGrpcFaults(fault, duration)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `averageDelay` | string | Average delay added to responses |
+| `delayVariation` | string | Variation around averageDelay |
+| `errorRate` | float | Fraction of requests returning errors (0.0-1.0) |
+| `statusCode` | integer | gRPC status code for errors |
+| `statusMessage` | string | gRPC status message for errors |
+| `port` | integer | Target port |
+| `exclude` | string | Service method to exclude |
+
+```javascript
+disruptor.injectGrpcFaults({
+  averageDelay: '500ms',
+  errorRate: 0.2,
+  statusCode: 14,       // UNAVAILABLE
+  statusMessage: 'injected fault',
+}, '60s');
+```
+
+### terminatePods(options)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `count` | integer | Number of pods to terminate |
+| `interval` | string | Wait time between terminations |
+
+```javascript
+disruptor.terminatePods({
+  count: 1,
+  interval: '30s',
+});
+```
+
+## PodDisruptor
+
+Targets pods by label selector.
+
+```javascript
+import { PodDisruptor } from 'k6/x/disruptor';
+
+const disruptor = new PodDisruptor({
+  namespace: 'default',
+  select: {
+    labels: { app: 'my-app', version: 'v1' },
+  },
+});
+```
+
+Same methods as ServiceDisruptor: `injectHTTPFaults()`, `injectGrpcFaults()`, `terminatePods()`.
+
+Additional method:
+
+### targets()
+
+Returns list of target pod names:
+
+```javascript
+const pods = disruptor.targets();
+console.log(`Targeting ${pods.length} pods:`, pods);
+```
+
+## Kubernetes Requirements
+
+### RBAC
+
+The k6 pod needs permissions to manage target pods:
+
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: k6-disruptor
+rules:
+  - apiGroups: [""]
+    resources: ["pods", "pods/ephemeralcontainers"]
+    verbs: ["get", "list", "patch", "update", "delete"]
+  - apiGroups: [""]
+    resources: ["services"]
+    verbs: ["get", "list"]
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["get", "list"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: k6-disruptor-binding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: k6-disruptor
+subjects:
+  - kind: ServiceAccount
+    name: default
+    namespace: k6-test
+```
+
+### Running in Cluster
+
+For xk6-disruptor to work, k6 must run **inside the Kubernetes cluster** (not locally). Options:
+- k6-operator TestRun
+- Pod with k6 + xk6-disruptor image
+- CI/CD runner with cluster access
+
+## Fault Duration vs Test Duration
+
+```
+Test Duration:     |========================== 5 min ============================|
+Fault Duration:    |======== 2 min ========|
+                   ↑ Fault injected         ↑ Fault removed    ↑ Test ends
+
+Phases:            |--- Chaos Phase ------|--- Recovery Phase ---|
+```
+
+Set fault duration shorter than test duration to observe both degradation and recovery.