@soleur/opencode 0.0.0-stage → 1.0.0

package/skills/dspy-ruby/references/providers.md

@@ -0,0 +1,335 @@
+# DSPy.rb LLM Providers
+
+## Supported Providers
+
+DSPy.rb provides unified support across multiple LLM providers through adapter gems that automatically load when installed.
+
+### Provider Overview
+
+- **OpenAI**: GPT-4, GPT-4o, GPT-4o-mini, GPT-3.5-turbo
+- **Anthropic**: Claude Opus 5.5, Sonnet 5.5, Haiku 5.5
+- **Google Gemini**: Gemini 1.5 Pro, Gemini 1.5 Flash, other versions
+- **Ollama**: Local model support via OpenAI compatibility layer
+- **OpenRouter**: Unified multi-provider API for 200+ models
+
+## Configuration
+
+### Basic Setup
+
+```ruby
+require 'dspy'
+
+DSPy.configure do |c|
+  c.lm = DSPy::LM.new('provider/model-name', api_key: ENV['API_KEY'])
+end
+```
+
+### OpenAI Configuration
+
+**Required gem**: `dspy-openai`
+
+```ruby
+DSPy.configure do |c|
+  # GPT-4o Mini (recommended for development)
+  c.lm = DSPy::LM.new('openai/gpt-4o-mini', api_key: ENV['OPENAI_API_KEY'])
+
+  # GPT-4o (more capable)
+  c.lm = DSPy::LM.new('openai/gpt-4o', api_key: ENV['OPENAI_API_KEY'])
+
+  # GPT-4 Turbo
+  c.lm = DSPy::LM.new('openai/gpt-4-turbo', api_key: ENV['OPENAI_API_KEY'])
+end
+```
+
+**Environment variable**: `OPENAI_API_KEY`
+
+### Anthropic Configuration
+
+**Required gem**: `dspy-anthropic`
+
+```ruby
+DSPy.configure do |c|
+  # Claude Opus 5.5 (most intelligent, best for agents and coding)
+  c.lm = DSPy::LM.new('anthropic/claude-opus-5-5',
+    api_key: ENV['ANTHROPIC_API_KEY'])
+
+  # Claude Sonnet 5.5 (best speed/intelligence balance)
+  c.lm = DSPy::LM.new('anthropic/claude-sonnet-5-5',
+    api_key: ENV['ANTHROPIC_API_KEY'])
+
+  # Claude Haiku 5.5 (fastest, cost-effective)
+  c.lm = DSPy::LM.new('anthropic/claude-haiku-5-5',
+    api_key: ENV['ANTHROPIC_API_KEY'])
+end
+```
+
+**Environment variable**: `ANTHROPIC_API_KEY`
+
+### Google Gemini Configuration
+
+**Required gem**: `dspy-gemini`
+
+```ruby
+DSPy.configure do |c|
+  # Gemini 1.5 Pro (most capable)
+  c.lm = DSPy::LM.new('gemini/gemini-1.5-pro',
+    api_key: ENV['GOOGLE_API_KEY'])
+
+  # Gemini 1.5 Flash (faster, cost-effective)
+  c.lm = DSPy::LM.new('gemini/gemini-1.5-flash',
+    api_key: ENV['GOOGLE_API_KEY'])
+end
+```
+
+**Environment variable**: `GOOGLE_API_KEY` or `GEMINI_API_KEY`
+
+### Ollama Configuration
+
+**Required gem**: None (uses OpenAI compatibility layer)
+
+```ruby
+DSPy.configure do |c|
+  # Local Ollama instance
+  c.lm = DSPy::LM.new('ollama/llama3.1',
+    base_url: 'http://localhost:11434')
+
+  # Other Ollama models
+  c.lm = DSPy::LM.new('ollama/mistral')
+  c.lm = DSPy::LM.new('ollama/codellama')
+end
+```
+
+**Note**: Ensure Ollama is running locally: `ollama serve`
+
+### OpenRouter Configuration
+
+**Required gem**: `dspy-openai` (uses OpenAI adapter)
+
+```ruby
+DSPy.configure do |c|
+  # Access 200+ models through OpenRouter
+  c.lm = DSPy::LM.new('openrouter/anthropic/claude-sonnet-5-5',
+    api_key: ENV['OPENROUTER_API_KEY'],
+    base_url: 'https://openrouter.ai/api/v1')
+
+  # Other examples
+  c.lm = DSPy::LM.new('openrouter/google/gemini-pro')
+  c.lm = DSPy::LM.new('openrouter/meta-llama/llama-3.1-70b-instruct')
+end
+```
+
+**Environment variable**: `OPENROUTER_API_KEY`
+
+## Provider Compatibility Matrix
+
+### Feature Support
+
+| Feature | OpenAI | Anthropic | Gemini | Ollama |
+|---------|--------|-----------|--------|--------|
+| Structured Output | ✅ | ✅ | ✅ | ✅ |
+| Vision (Images) | ✅ | ✅ | ✅ | ⚠️ Limited |
+| Image URLs | ✅ | ❌ | ❌ | ❌ |
+| Tool Calling | ✅ | ✅ | ✅ | Varies |
+| Streaming | ❌ | ❌ | ❌ | ❌ |
+| Function Calling | ✅ | ✅ | ✅ | Varies |
+
+**Legend**: ✅ Full support | ⚠️ Partial support | ❌ Not supported
+
+### Vision Capabilities
+
+**Image URLs**: Only OpenAI supports direct URL references. For other providers, load images as base64 or from files.
+
+```ruby
+# OpenAI - supports URLs
+DSPy::Image.from_url("https://example.com/image.jpg")
+
+# Anthropic, Gemini - use file or base64
+DSPy::Image.from_file("path/to/image.jpg")
+DSPy::Image.from_base64(base64_data, mime_type: "image/jpeg")
+```
+
+**Ollama**: Limited multimodal functionality. Check specific model capabilities.
+
+## Advanced Configuration
+
+### Custom Parameters
+
+Pass provider-specific parameters during configuration:
+
+```ruby
+DSPy.configure do |c|
+  c.lm = DSPy::LM.new('openai/gpt-4o',
+    api_key: ENV['OPENAI_API_KEY'],
+    temperature: 0.7,
+    max_tokens: 2000,
+    top_p: 0.9
+  )
+end
+```
+
+### Multiple Providers
+
+Use different models for different tasks:
+
+```ruby
+# Fast model for simple tasks
+fast_lm = DSPy::LM.new('openai/gpt-4o-mini', api_key: ENV['OPENAI_API_KEY'])
+
+# Powerful model for complex tasks
+powerful_lm = DSPy::LM.new('anthropic/claude-sonnet-5-5',
+  api_key: ENV['ANTHROPIC_API_KEY'])
+
+# Use different models in different modules
+class SimpleClassifier < DSPy::Module
+  def initialize
+    super
+    DSPy.configure { |c| c.lm = fast_lm }
+    @predictor = DSPy::Predict.new(SimpleSignature)
+  end
+end
+
+class ComplexAnalyzer < DSPy::Module
+  def initialize
+    super
+    DSPy.configure { |c| c.lm = powerful_lm }
+    @predictor = DSPy::ChainOfThought.new(ComplexSignature)
+  end
+end
+```
+
+### Per-Request Configuration
+
+Override configuration for specific predictions:
+
+```ruby
+predictor = DSPy::Predict.new(MySignature)
+
+# Use default configuration
+result1 = predictor.forward(input: "data")
+
+# Override temperature for this request
+result2 = predictor.forward(
+  input: "data",
+  config: { temperature: 0.2 }  # More deterministic
+)
+```
+
+## Cost Optimization
+
+### Model Selection Strategy
+
+1. **Development**: Use cheaper, faster models (gpt-4o-mini, claude-haiku-5-5, gemini-1.5-flash)
+2. **Production Simple Tasks**: Continue with cheaper models if quality is sufficient
+3. **Production Complex Tasks**: Upgrade to more capable models (gpt-4o, claude-sonnet-5-5, gemini-1.5-pro)
+4. **Local Development**: Use Ollama for privacy and zero API costs
+
+### Example Cost-Conscious Setup
+
+```ruby
+# Development environment
+if Rails.env.development?
+  DSPy.configure do |c|
+    c.lm = DSPy::LM.new('ollama/llama3.1')  # Free, local
+  end
+elsif Rails.env.test?
+  DSPy.configure do |c|
+    c.lm = DSPy::LM.new('openai/gpt-4o-mini',  # Cheap for testing
+      api_key: ENV['OPENAI_API_KEY'])
+  end
+else  # production
+  DSPy.configure do |c|
+    c.lm = DSPy::LM.new('anthropic/claude-sonnet-5-5',
+      api_key: ENV['ANTHROPIC_API_KEY'])
+  end
+end
+```
+
+## Provider-Specific Best Practices
+
+### OpenAI
+
+- Use `gpt-4o-mini` for development and simple tasks
+- Use `gpt-4o` for production complex tasks
+- Best vision support including URL loading
+- Excellent function calling capabilities
+
+### Anthropic
+
+- Claude Sonnet 5.5 offers the best speed/intelligence balance
+- Claude Opus 5.5 is the most intelligent for agents and coding
+- Excellent for complex reasoning and analysis
+- Strong safety features and helpful outputs
+- Requires base64 for images (no URL support)
+
+### Google Gemini
+
+- Gemini 1.5 Pro for complex tasks, Flash for speed
+- Strong multimodal capabilities
+- Good balance of cost and performance
+- Requires base64 for images
+
+### Ollama
+
+- Best for privacy-sensitive applications
+- Zero API costs
+- Requires local hardware resources
+- Limited multimodal support depending on model
+- Good for development and testing
+
+## Troubleshooting
+
+### API Key Issues
+
+```ruby
+# Verify API key is set
+if ENV['OPENAI_API_KEY'].nil?
+  raise "OPENAI_API_KEY environment variable not set"
+end
+
+# Test connection
+begin
+  DSPy.configure { |c| c.lm = DSPy::LM.new('openai/gpt-4o-mini',
+    api_key: ENV['OPENAI_API_KEY']) }
+  predictor = DSPy::Predict.new(TestSignature)
+  predictor.forward(test: "data")
+  puts "✅ Connection successful"
+rescue => e
+  puts "❌ Connection failed: #{e.message}"
+end
+```
+
+### Rate Limiting
+
+Handle rate limits gracefully:
+
+```ruby
+def call_with_retry(predictor, input, max_retries: 3)
+  retries = 0
+  begin
+    predictor.forward(input)
+  rescue RateLimitError => e
+    retries += 1
+    if retries < max_retries
+      sleep(2 ** retries)  # Exponential backoff
+      retry
+    else
+      raise
+    end
+  end
+end
+```
+
+### Model Not Found
+
+Ensure the correct gem is installed:
+
+```bash
+# For OpenAI
+gem install dspy-openai
+
+# For Anthropic
+gem install dspy-anthropic
+
+# For Gemini
+gem install dspy-gemini
+```

package/skills/eval-harness/README.md

@@ -0,0 +1,194 @@
+# eval-harness
+
+A [promptfoo](https://www.promptfoo.dev) harness that empirically measures whether a Soleur skill
+or agent edit improves classification behavior. v1 ships two targets — `soleur:go` routing accuracy
+and ticket-triage P-level accuracy — adapting the [ponytail](https://github.com/DietrichGebert/ponytail)
+(MIT) benchmark methodology.
+
+## Prerequisites
+
+- Node `>=22.22` (the worktree ships v22.22.1) — no dependency is added to any `package.json`;
+  the harness runs via `npx promptfoo`.
+- `ANTHROPIC_API_KEY` in the environment (promptfoo's `anthropic:messages:*` providers read it).
+  In this repository, take it from Doppler `ci`: `doppler run -p soleur -c ci -- <command>`. That
+  key lives in the spend-capped `soleur-ci-eval` Console workspace ($100/month), so a grid can
+  never drain the balance production draws on. Never run a grid under a `prd*` config (#8505).
+
+## Reproduce
+
+```bash
+cd plugins/soleur/skills/eval-harness
+
+# 1. Refresh the model IDs from the TS registry (writes models.generated.json).
+bash scripts/gen-models.sh
+
+# 2. (No-spend) sanity-check the configs — zero API calls.
+npx promptfoo validate config -c promptfooconfig-go-routing.yaml
+npx promptfoo validate config -c promptfooconfig-ticket-triage.yaml
+
+# 3. Run a target (SPENDS — see cost below). `--repeat 3` runs each cell 3×.
+npx promptfoo eval -c promptfooconfig-go-routing.yaml --repeat 3
+npx promptfoo eval -c promptfooconfig-ticket-triage.yaml --repeat 3
+
+# 4. Inspect results in the browser.
+npx promptfoo view
+```
+
+## Projection-regen (run on any source-block edit)
+
+The skill-arm prompts are mechanical projections of the production classifier blocks, not
+hand-copies. After editing the `/go` routing table (`plugins/soleur/commands/go.md`) or the
+ticket-triage rubric (`plugins/soleur/agents/support/ticket-triage.md`), regenerate the prompts:
+
+```bash
+cd plugins/soleur/skills/eval-harness
+node scripts/gen-skill-prompt.cjs --all       # rewrites prompts/go-skill.txt + prompts/triage-skill.txt
+bash test/extract-block.test.sh               # AC4: generated projection == committed (no API)
+```
+
+## Gate mode — validation-gated block edits (reproduce)
+
+```bash
+cd plugins/soleur/skills/eval-harness
+
+# Is a file a gated classifier source? (lookup-only, no API)
+node scripts/eval-gate.cjs --check ../../commands/go.md
+
+# What would a gate run cost? (skill-arm-only estimate, no API)
+node scripts/eval-gate.cjs --dry-run --target go-routing --repeat 5
+
+# Gate an edit (SPENDS): current block on disk vs the candidate edit; the targeted case must pass
+# and the corpus must not regress. Append the synthesized golden task to the corpus only on accept.
+node scripts/eval-gate.cjs --target go-routing \
+  --candidate-file /path/to/edited-go.md \
+  --target-task '{"vars":{"input":"...","golden_label":"fix"}}' \
+  --repeat 5 --append-on-accept
+```
+
+The verdict math is the pure [scripts/verdict.cjs](./scripts/verdict.cjs) (`computeVerdict`); the gate
+fails closed (non-zero exit + NOT accept on any error). See [SKILL.md](./SKILL.md) §"Gate mode" and
+ADR-069 for the decision and the projection precondition.
+
+> **Use `validate config`, never bare `validate` or `validate target`** — the latter two spend API
+> credits. `validate config` is config-only and free.
+>
+> **`--repeat N` is a CLI flag, not a config key.** A `repeat:` key in `promptfooconfig.yaml` is
+> silently ignored (each cell runs once). Pass `--repeat 3` on the command line to get N runs per
+> cell for the median.
+
+## Per-run cost
+
+Each target's grid is **2 arms × 3 models × (its golden tasks) × `--repeat`**. At the current task
+counts and `--repeat 3`: ≈ **126 API calls** for go-routing (7 tasks) and ≈ **108** for
+ticket-triage (6 tasks) — ≈ **230 to run both**. One arm runs on Opus, so cost is dominated by the
+Opus cells, but outputs are single tokens so per-call cost is small (the first full run was well
+under $1). This is why the harness is **opt-in and manual** and is NOT wired into per-PR CI. See the
+`<decision_gate>` in [SKILL.md](./SKILL.md) for the billing disclosure.
+
+Before any paid grid that asks for free-text generation, smoke it with
+`--filter-first-n 1 --repeat 1 --no-cache` and read each row's output length and `finishReason`.
+Opus 5.5 and Sonnet 5.5 think by default, so a small `ANTHROPIC_MAX_TOKENS` (300 measured) spends the whole
+budget on thinking and returns empty text, which a lexical scorer reads as compliant (#8290 needed
+3000). Estimate spend from a measured prompt size rather than from file bytes (the #8290 estimate ran
+39% low). promptfoo prints no progress when it isn't writing to a terminal, so poll
+`sqlite3 ~/.promptfoo/promptfoo.db "select count(*) from eval_results where eval_id like '<id>%'"`.
+
+## First recorded delta (2026-06-15, `--repeat 3`, opus-4-8 / sonnet-4-6 / haiku-4-5)
+
+The baseline run that proved the rig pays off (point-in-time; model updates will shift these):
+
+| Target | Skill arm | Baseline arm | Delta |
+|--------|-----------|--------------|-------|
+| `soleur:go` routing | 100% (63/63) | 81% (51/63) | **+19 pts** |
+| ticket-triage P-level | 94% (51/54) | 89% (48/54) | **+6 pts** |
+
+Read: the `/go` routing table produces a clean, consistent advantage across all three models (the
+Sonnet baseline scored only 71% on the routing-rule cases → 100% with the table). The triage rubric
+adds a smaller, real delta — priority is more inferable from raw impact than routing is from intent.
+The takeaway for expanding the harness: it discriminates routing-type classifiers strongly, prose
+rubrics modestly, so prioritize new targets by how much the rule adds beyond the label names.
+
+## Reading the baseline-vs-skill delta
+
+Each run produces, per (arm × model) cell, the **classification-correct rate** recorded by the
+measurement assert (mean of the per-task 1.0/0.0 scores; promptfoo has no native median, so read
+the aggregate score promptfoo reports per cell, or compute the median across the `--repeat` runs from
+the JSON output).
+
+> **delta = (skill-arm correct rate) − (baseline-arm correct rate)**
+
+A **positive delta** is the evidence that the classifier prose (the `/go` routing table, the triage
+rubric) *produces* the routing/triage behavior — not merely that the text is present. A delta near
+zero means the model classifies just as well without the prose (the prose is not adding signal for
+these tasks). The **gate assert** is orthogonal: it fails any cell whose emitted label is malformed
+or outside the closed enum, independent of correctness — a baseline arm that hallucinates a
+non-existent route is caught here even when its correct rate looks fine.
+
+> **Golden tasks must be *discriminating*, not just correct.** The first POC run used easy,
+> unambiguous tasks and the baseline arm scored ~89–100% — so the delta was ~0 and the harness
+> proved nothing. The discriminating tasks are the ones where the *rule* matters but the label
+> *name* alone misleads a label-only baseline: a live production outage that reads like a "fix"
+> (→ `incident`), a "close out the whole backlog" that reads like a "fix" (→ `drain`), a
+> data-deletion request that reads like "default" (→ `legal-threshold`), an `URGENT!!`-titled
+> cosmetic typo (→ `P3`), a "no new user can activate" with no scary keywords (→ `P1`). When adding
+> tasks, bias toward these adversarial cases — that is where the delta lives.
+
+To regression-check a future edit to `/go` or ticket-triage: run the relevant config before and
+after the edit and compare the skill-arm correct rate. A drop is a behavioral regression the prose
+change introduced.
+
+## Adding a new target (the additive recipe)
+
+The two-target v1 exists to prove this is cheap. To add target N:
+
+1. Add an enum SSOT `enums/<target>.json` (the closed label set).
+2. Add two arm prompts `prompts/<target>-skill.txt` (embeds the production classifier prose) and
+   `prompts/<target>-baseline.txt` (label set only, no rules). Each uses the `{{input}}` placeholder.
+3. Add golden tasks `tasks/<target>.jsonl` — `{"vars": {"input": "...", "golden_label": "..."}}` per
+   line, synthesized fixtures only.
+4. Add `promptfooconfig-<target>.yaml` mirroring the existing configs: `providers: file://models.generated.json`,
+   the two prompts, `defaultTest.vars.enum: file://enums/<target>.json`, and the two shared asserts
+   ([measure-classification.cjs](./scripts/measure-classification.cjs),
+   [gate-classification.cjs](./scripts/gate-classification.cjs)).
+
+The generic asserts and `models.generated.json` are reused unchanged — no new script is needed
+unless the target's output is not a single-token label (prose-output surfaces need an LLM-judge
+assert, deferred out of v1).
+
+**Multi-label surface? Gate its single-token output slice first.** A classifier that emits a
+*set* (e.g. `pdr-*` passive-domain-routing → which leaders to spawn) is multi-label and would
+need a set-membership assert. Before building one, check whether the surface also has a
+single-token output factor — a count, tier, or breadth — that captures most of the regression
+risk. `pdr`'s breadth factor is exactly the `lane-inference` enum (`procedural`/`single-domain`/
+`cross-domain`), so gating lane-inference covers the verifiable slice with zero new code and
+defers only the irreducible set factor. Half-covering a multi-label surface with a single-label
+harness ships false confidence — defer it whole instead. See
+`knowledge-base/project/learnings/2026-06-29-multi-label-classifier-gateable-core-is-its-single-token-output-slice.md`.
+
+## Archived measurement: `rule-phrasing` (B5, #8290)
+
+B5 measured positive-led vs prohibition-led phrasing of four always-loaded rules (648 calls,
+2026-09-21). The verdict was INCONCLUSIVE with limited instrument validity, and the machinery was
+archived rather than kept: see `knowledge-base/project/specs/archive/20260921-162911-feat-one-shot-8290-invocation-axis-budget-relief/b5-eval-results.md`
+for the record and the recovery command, and #8497 for the rerun conditions.
+
+## Files
+
+| Path | Role |
+|------|------|
+| [promptfooconfig-go-routing.yaml](./promptfooconfig-go-routing.yaml) | `/go` routing target config |
+| [promptfooconfig-ticket-triage.yaml](./promptfooconfig-ticket-triage.yaml) | ticket-triage target config |
+| [scripts/gen-models.sh](./scripts/gen-models.sh) | single-sources the 3 model IDs → `models.generated.json` |
+| [scripts/extract-block.cjs](./scripts/extract-block.cjs) | extracts a gated source block between sentinels (the projection seam) |
+| [scripts/gen-skill-prompt.cjs](./scripts/gen-skill-prompt.cjs) | projects a block into a skill-arm prompt (regenerate on source edit) |
+| [scripts/verdict.cjs](./scripts/verdict.cjs) | pure `computeVerdict` (no I/O) — the accept/reject seam |
+| [scripts/eval-gate.cjs](./scripts/eval-gate.cjs) | gate orchestrator (`--check`, `--dry-run`, real run, `--append-on-accept`) |
+| [gated-skills.json](./gated-skills.json) | registry of gated source blocks (source_file, markers, target, projected prompt) |
+| [scripts/parse-label.cjs](./scripts/parse-label.cjs) | shared label parser |
+| [scripts/measure-classification.cjs](./scripts/measure-classification.cjs) | MEASUREMENT assert (always passes, records the rate) |
+| [scripts/gate-classification.cjs](./scripts/gate-classification.cjs) | GATE assert (fails on out-of-enum label) |
+| [enums/go-routes.json](./enums/go-routes.json) · [enums/triage-levels.json](./enums/triage-levels.json) | closed label sets |
+| [prompts/](./prompts/go-skill.txt) | skill + baseline arm templates per target |
+| [tasks/go-routing.jsonl](./tasks/go-routing.jsonl) · [tasks/ticket-triage.jsonl](./tasks/ticket-triage.jsonl) | synthesized golden tasks |
+| [test/](./test/gen-models.test.sh) | deterministic `.test.sh` unit tests (no live LLM) |
+| [test/registry-completeness.test.sh](./test/registry-completeness.test.sh) | parity gate: `eval-gate:block` source markers ↔ `gated-skills.json` `block_id`s (DEDUP + set-equality + charset) |