@assertrx/dino 1.0.11 → 1.0.12

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@assertrx/dino",
-  "version": "1.0.11",
+  "version": "1.0.12",
   "description": "A friendly purple dinosaur pi package with a custom TUI header, theme, and Playwright skills.",
   "keywords": [
     "pi-package"

package/prompts/setup-promptfoo.md

@@ -0,0 +1,103 @@
+---
+description: Install promptfoo and create a runnable skeleton AI-evals project in the current workspace
+---
+
+Set up [promptfoo](https://www.promptfoo.dev/) and a runnable skeleton AI-evals project in the current workspace root. The goal is to end up with the promptfoo CLI available and a minimal `promptfoo/` folder with a working smoke eval config that passes offline (no API key required).
+
+## Steps
+
+1. **Inspect the workspace**.
+   - Use `bash` to list the current directory and check for `package.json`.
+   - Do not touch existing `promptfooconfig.yaml` files, `promptfoo/` folders, or CI pipelines that already reference promptfoo.
+
+2. **Check promptfoo availability**.
+   - Run `npx -y promptfoo@latest --version`.
+   - If it prints a version (0.70 or newer), note it and skip installation.
+
+3. **Install promptfoo if needed**. Prefer a local dev dependency so runs are reproducible:
+   - If `package.json` is missing, run `npm init -y` first.
+   - Run `npm install --save-dev promptfoo`.
+   - Verify with `npx promptfoo --version`.
+   - Fallback if npm install fails or the user prefers no dependency: rely on `npx -y promptfoo@latest` and document that usage in the README.
+
+4. **Load the promptfoo skill**.
+   - Use `read` to load the SKILL.md of the `promptfoo` skill from the package `skills/` directory, and follow the reference files it points to (`reference/assert-types.md`, `reference/providers.md`) when needed, so the scaffolded files follow the current schema and idioms.
+
+5. **Create the project structure**.
+   - Keep it minimal and idiomatic:
+     ```
+     promptfoo/
+       promptfooconfig.yaml
+       README.md
+     ```
+   - (`promptfoo/prompt.txt` is optional — see step 6.)
+
+6. **Create the smoke eval config**.
+   - Write `promptfoo/promptfooconfig.yaml` with a config that runs **offline and passes without any API key** by using the `echo` provider:
+     ```yaml
+     description: "AI evals — smoke suite (offline)"
+
+     prompts:
+       # Inline template keeps the skeleton dependency-free; the file://
+       # form also works once you add prompt files:
+       # - file://prompts/system.txt
+       - "You are a QA assistant. Explain {{topic}} in one sentence."
+
+     providers:
+       # Echo provider: returns the rendered prompt — zero cost, offline.
+       # Swap in a real provider when ready (needs the matching env var):
+       # - anthropic:claude-3-5-haiku-latest   # ANTHROPIC_API_KEY
+       # - openai:gpt-4o-mini                  # OPENAI_API_KEY
+       - echo
+
+     defaultTest:
+       assert:
+         - type: latency
+           threshold: 2000
+
+     tests:
+       - description: "Template renders the variable"
+         vars:
+           topic: load testing
+         assert:
+           - type: contains
+             value: load testing
+     ```
+   - Write the prompt template `promptfoo/prompt.txt` next to the config for users who prefer file-based prompts (referenced as `file://prompt.txt`); the inline default keeps the skeleton working regardless.
+
+7. **Create the README**.
+   - Write `promptfoo/README.md` covering:
+     - How to run: `npx promptfoo eval -c promptfoo/promptfooconfig.yaml` (or the npm script).
+     - How to view results in the web UI: `npx promptfoo view`.
+     - Which env vars to set for the commented-out real providers (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`) — never hardcode keys.
+     - One-line pointer that more eval actions (test generation) exist in the Dino package.
+
+8. **Add convenience scripts**.
+   - Use `edit` to add scripts to `package.json` (create it with `npm init -y` first if needed):
+     ```json
+     {
+       "eval:smoke": "promptfoo eval -c promptfoo/promptfooconfig.yaml",
+       "eval:view": "promptfoo view"
+     }
+     ```
+   - Do not duplicate scripts that already exist.
+
+9. **Verify the setup**.
+   - Run `npx promptfoo --version` and report the version.
+   - Run `npx promptfoo eval -c promptfoo/promptfooconfig.yaml` — the echo provider makes this offline-safe and it must end with all tests passing and exit code 0. Report the pass/fail counts.
+   - Report the final promptfoo version and the list of created files.
+
+## Rules
+
+- Do not overwrite existing user files unless they are missing or clearly incomplete.
+- Keep the setup minimal and idiomatic; do not add redteam configs, CI pipelines, or cloud integrations unless asked.
+- Never hardcode secrets or API keys in configs; use environment variables exclusively.
+- The smoke config must run offline via the `echo` provider — real providers stay commented out until the user sets keys.
+- Use `edit` for precise changes to `package.json`.
+- Use `write` only for new files or complete rewrites of files you created.
+
+After completing the setup, summarize:
+- Which files were created or modified.
+- The promptfoo version installed.
+- The smoke eval result (tests passed/failed) and any errors encountered.
+- How to switch from the echo provider to a real LLM provider.

package/scripts/api-server.mjs

@@ -93,6 +93,11 @@ const PROMPT_REGISTRY = {
     params: [],
     description: "Install k6 and create a runnable skeleton load-testing project in the workspace",
   },
+  "setup-promptfoo": {
+    file: "setup-promptfoo.md",
+    params: [],
+    description: "Install promptfoo and create a runnable skeleton AI-evals project in the workspace",
+  },
   "generate-k6-test-from-endpoint": {
     file: "generate-k6-test-from-endpoint.md",
     params: [

package/skills/promptfoo/SKILL.md

@@ -0,0 +1,121 @@
+---
+name: promptfoo
+description: Use when writing, running, or debugging promptfoo AI evaluations — promptfooconfig.yaml structure, providers (OpenAI/Anthropic/HTTP/echo), test cases with vars, assert types (contains, is-json, javascript, similar, llm-rubric), CLI usage (eval, view, JSON output), thresholds, and CI integration. Use when the user asks about LLM evals, prompt evaluation, model output testing, or setting up a promptfoo project.
+---
+
+# Promptfoo — LLM / Prompt Evaluations
+
+Author, run, and interpret [promptfoo](https://www.promptfoo.dev/) evaluations. Promptfoo evaluates LLM prompts, agents, and model-backed endpoints against test cases with deterministic and model-graded assertions, producing a pass/fail result like a test runner.
+
+## When to Use Promptfoo
+
+| Goal | Approach |
+|------|----------|
+| Verify prompt/model changes don't regress outputs | Eval suite with deterministic asserts on known inputs |
+| Compare prompts or models side by side | Multiple `prompts` or `providers` in one config — promptfoo renders a matrix |
+| Test subjective quality (tone, faithfulness) | `llm-rubric` assert (author it with the **write-judge-prompt** skill) |
+| Test an HTTP endpoint that wraps a model | `http` provider hitting the real endpoint |
+| Sanity-check the harness itself, no API needed | `echo` provider — instant, offline, zero cost |
+| Find vulnerable/unsafe outputs | `npx promptfoo redteam` (separate command, own config) |
+
+## Project Layout
+
+```
+promptfoo/
+  promptfooconfig.yaml   # the eval config: prompts + providers + tests
+  prompts/
+    system.txt           # prompt templates ({{var}} placeholders) — or inline
+  tests/                 # optional: split test cases into YAML/CSV/JSON files
+  README.md              # how to run, env vars needed
+```
+
+A single `promptfooconfig.yaml` is enough for a skeleton project; split prompts/tests into files only when the config grows.
+
+## Config Anatomy (promptfooconfig.yaml)
+
+```yaml
+description: "Customer support agent — smoke evals"
+
+# What to evaluate. One or more prompt templates; {{vars}} come from tests.
+prompts:
+  - file://promptfoo/prompts/system.txt
+  # or inline:
+  # - "Translate the following to French: {{text}}"
+
+# Who answers. Multiple providers = comparison matrix.
+providers:
+  - openai:gpt-4o-mini          # needs OPENAI_API_KEY
+  - anthropic:claude-3-5-haiku-latest  # needs ANTHROPIC_API_KEY
+  # Local/zero-cost provider for harness checks:
+  # - echo
+
+# Test cases: inputs + expected behavior.
+tests:
+  - description: "Greeting is friendly"
+    vars:
+      message: "hi there"
+    assert:
+      - type: contains
+        value: "hello"
+      - type: llm-rubric
+        value: "The reply is polite and offers help"
+
+# Assertions applied to every test (defaults; per-test asserts extend these).
+defaultTest:
+  assert:
+    - type: latency
+      threshold: 5000   # ms
+```
+
+Full assert reference: [reference/assert-types.md](reference/assert-types.md). Provider reference: [reference/providers.md](reference/providers.md).
+
+## Running Evals
+
+```bash
+# Run the default config (promptfooconfig.yaml in cwd)
+npx promptfoo@latest eval
+
+# Run a specific config, fail the command on failed asserts (default)
+npx promptfoo@latest eval -c promptfoo/promptfooconfig.yaml
+
+# Machine-readable results (for CI or parsing)
+npx promptfoo@latest eval -c promptfoo/promptfooconfig.yaml --output json
+
+# Interactive web viewer of the last results
+npx promptfoo@latest view
+```
+
+- **Exit code**: `0` when all asserts pass, non-zero when any test fails or the config is invalid — safe for CI.
+- **Cost control**: `--repeat 1` (default), keep test counts small, prefer cheap models while iterating.
+- **Concurrency**: defaults are conservative; raise via `--max-concurrency N` for larger suites.
+
+## Test-Design Rules
+
+1. **Deterministic first.** Use `contains`, `is-json`, `javascript`, and `similar` before model-graded asserts — they are reproducible, free, and debuggable.
+2. **Rubric only for the subjective.** For `llm-rubric`, write the judge criteria with the **write-judge-prompt** skill, and calibrate it with **validate-evaluator** before trusting results.
+3. **Small, sharp suites.** 5–20 targeted cases beat 200 vague ones. Generate bulk synthetic variants with **generate-synthetic-data** only when you need coverage/robustness checks.
+4. **No secrets in configs.** API keys come from env (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, …) — never commit them.
+5. **Thresholds over vibes.** Set `defaultTest` latency/cost guards so regressions fail loudly even when content asserts pass.
+
+## CI Integration (GitHub Actions)
+
+```yaml
+- name: Run prompt evals
+  env:
+    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+  run: npx promptfoo@latest eval -c promptfoo/promptfooconfig.yaml --output json > results.json
+```
+
+## Interpreting Results
+
+- Each row = one test × (prompt × provider). Red cells list the exact assert that failed with actual vs expected — read that first.
+- For `llm-rubric` failures, inspect the judge's reason in the viewer before assuming the model is wrong; a mis-calibrated judge is a common failure mode (see **eval-audit** skill).
+- Flaky content asserts (`contains` on chatty models) → relax to `contains-any`, `similar` with a threshold, or a better-scoped rubric.
+
+## Related Skills in This Package
+
+- **write-judge-prompt** — author the `llm-rubric` criteria.
+- **validate-evaluator** — calibrate judges against human labels before trusting scores.
+- **eval-audit** — audit an existing eval suite for missing error analysis, vanity metrics, etc.
+- **generate-synthetic-data** — bulk-generate test inputs across behavioral dimensions.
+- **error-analysis** — categorize failures after a run to decide what to fix first.

package/skills/promptfoo/reference/assert-types.md

@@ -0,0 +1,74 @@
+# Promptfoo Assert Types Reference
+
+Assertions live under each test's `assert:` list (and `defaultTest.assert:` for suite-wide defaults). Every assert evaluates to pass/fail against the provider output. Prefer deterministic types; use model-graded types only for subjective criteria.
+
+## Deterministic (reproducible, free)
+
+| Type | What it checks | Key fields |
+|------|----------------|-----------|
+| `contains` | Output includes a substring | `value` |
+| `not-contains` | Output must not include a substring | `value` |
+| `contains-any` | Output includes at least one of several substrings | `value: [..]` |
+| `contains-all` | Output includes all of several substrings | `value: [..]` |
+| `regex` | Output matches a pattern | `value` (regex string) |
+| `equals` | Exact match (trim/normalize first) | `value` |
+| `is-json` | Output parses as JSON; optionally validates a JSON schema | `value` (JSON schema) |
+| `javascript` | Custom predicate over output/context | `value` (JS expression returning boolean/number; `output`, `context` available) |
+| `python` | Same, in Python | `value` (code) |
+| `similar` | Embedding similarity to expected text | `value`, `threshold` (0–1, default 0.75) |
+| `latency` | Response time guard | `threshold` (ms) |
+| `cost` | Per-call cost guard | `threshold` (USD) |
+| `contains-sql` / `is-valid-sql` / `is-valid-function-call` / `is-valid-openai-function-call` / `is-valid-url` / `no-refusal` / `perplexity` / `perplexity-score` | Domain-specific structural checks | varies |
+
+## Model-graded (subjective criteria — judge LLM involved)
+
+| Type | What it checks | Notes |
+|------|----------------|-------|
+| `llm-rubric` | Output satisfies natural-language criteria | Judge prompt in `value`; author it with the **write-judge-prompt** skill. Optional `rubricPrompt` to override the default judge prompt template |
+| `factuality` | Output is factually consistent with a reference | `value` = ground truth, `subject` = original input |
+| `answer-relevance` | Output addresses the input | no extra fields |
+| `context-recall` / `context-relevance` / `context-faithfulness` | RAG quality: retrieved context vs question/answer | see **evaluate-rag** skill |
+| `classifier` | Label the output with a model-chosen category | `value` = list of allowed labels |
+
+## Grading Model
+
+Model-graded asserts run on a grader LLM. Configure it suite-wide:
+
+```yaml
+# Use a different model for judging than for the system under test
+evaluateOptions:
+  # or per-assert via the assert's own provider field
+providers: []
+```
+
+Simplest per-assert override:
+
+```yaml
+- type: llm-rubric
+  value: "Answer states the refund window and cites the policy doc."
+  # provider: anthropic:claude-3-5-haiku-latest   # optional judge override
+```
+
+## Weighted & Threshold Asserts
+
+```yaml
+tests:
+  - description: "Balanced checks"
+    vars: { q: "what is your return policy" }
+    assert:
+      - type: contains
+        value: "30 days"
+        weight: 2          # counts double in the score
+      - type: llm-rubric
+        value: "Polite and on-topic"
+        weight: 1
+    threshold: 0.8         # test passes only if score >= 0.8
+```
+
+## Suite-wide Pass Threshold
+
+```yaml
+# Fail the whole eval if less than 90% of tests pass
+tests:
+  - threshold: 0.9
+```

package/skills/promptfoo/reference/providers.md

@@ -0,0 +1,70 @@
+# Promptfoo Providers Reference
+
+Providers answer prompts. List one or more under `providers:` — multiple entries render a comparison matrix across all tests.
+
+## Model Providers
+
+```yaml
+providers:
+  - openai:gpt-4o-mini                 # needs OPENAI_API_KEY
+  - anthropic:claude-3-5-haiku-latest  # needs ANTHROPIC_API_KEY
+  - bedrock:us.anthropic.claude-3-5-sonnet-20241022-v2:0
+  - azureopenai:gpt-4o:my-deployment   # needs AZURE_OPENAI_API_BASE etc.
+  - google:gemini-1.5-flash            # needs GOOGLE_API_KEY
+  - ollama:chat:llama3.1               # local, no key
+```
+
+Custom model params (temperature, max tokens) use the long form:
+
+```yaml
+providers:
+  - id: anthropic:claude-3-5-haiku-latest
+    config:
+      temperature: 0
+      max_tokens: 512
+```
+
+## HTTP Provider (test any endpoint that wraps an LLM)
+
+```yaml
+providers:
+  - id: http
+    config:
+      url: https://api.example.com/v1/answer
+      method: POST
+      headers:
+        Content-Type: application/json
+        Authorization: Bearer {{ENV.API_TOKEN}}
+      body:
+        model: support-agent
+        messages: [{ role: user, content: "{{prompt}}" }]
+      transformResponse: json.response   # JSONPath-ish pick of the text to assert on
+```
+
+`{{prompt}}` is the rendered prompt template; `{{ENV.NAME}` reads environment variables — no secrets in the config file.
+
+## Echo Provider (offline harness checks)
+
+Returns the rendered prompt verbatim — no network, no cost. Use it to validate the config, wiring, and deterministic asserts before spending tokens (note: plain `- echo`, not `- echo:text` — the colon shorthand was removed in recent versions):
+
+```yaml
+providers:
+  - echo
+```
+
+## File/Script Providers
+
+- `file://path/to/provider.py` or `.js` — full control (custom agent, chain, RAG pipeline); export a `CallApi(prompt, context)` function.
+- Python file providers let you wrap internal SDKs not reachable over HTTP.
+
+## Environment Variables
+
+| Provider | Variable(s) |
+|----------|-------------|
+| OpenAI | `OPENAI_API_KEY` |
+| Anthropic | `ANTHROPIC_API_KEY` |
+| Google | `GOOGLE_API_KEY` / `GEMINI_API_KEY` |
+| Azure OpenAI | `AZURE_OPENAI_API_BASE`, `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_DEPLOYMENT_NAME` |
+| HTTP | anything referenced via `{{ENV.*}}` |
+
+Load from `.env` files or the shell; never commit keys.