@rudderhq/agent-runtime-gemini-local 0.2.1 → 0.2.2-canary.0

package/skills/conversation-to-skill/references/compatibility.md

@@ -0,0 +1,36 @@
+# Host Compatibility
+
+This skill is designed to work across multiple agent hosts. The workflow stays
+the same, but some mechanics change.
+
+## Capability Matrix
+
+| Host | Draft skill | Run evals | Parallel baseline | Description tuning | Packaging |
+|------|-------------|-----------|-------------------|--------------------|-----------|
+| Codex | Yes | Yes, judged via `codex exec` | Usually yes | Yes | Yes |
+| Claude Code | Yes | Yes, observed via `claude -p` | Yes | Yes | Yes |
+| Claude.ai | Yes | Manual/serial | No | Usually no | Yes |
+| Generic shell agent | Yes | Yes if a CLI exists | Depends | Depends | Yes |
+
+## Important Differences
+
+- **Claude Code**
+  - `scripts/run_eval.py --backend claude` measures real observed triggering.
+  - This is the highest-fidelity description benchmark.
+
+- **Codex**
+  - `scripts/run_eval.py --backend codex` measures judged routing, not native
+    skill invocation. It is still useful for testing whether the description
+    makes the intended use cases obvious.
+  - Treat Codex trigger numbers as a proxy, not ground truth.
+
+- **Claude.ai or chat-only hosts**
+  - Skip automated trigger benchmarks unless you have a shell + CLI bridge.
+  - Do serial manual evals and focus on qualitative output review.
+
+## Recommended Defaults
+
+- If you are improving instructions, tool choices, examples, or bundled scripts:
+  qualitative evals matter most.
+- If you are improving trigger phrasing:
+  prefer `--backend claude` when available, otherwise `--backend codex`.

package/skills/conversation-to-skill/references/description-optimization.md

@@ -0,0 +1,113 @@
+# Description Optimization
+
+Use this reference when the skill itself is already decent but its frontmatter
+description may under-trigger or over-trigger.
+
+The description is the primary routing surface.
+Treat optimization as a real evaluation problem, not copy editing.
+
+## When To Use This
+
+Use description optimization when:
+
+- the user asks to improve triggering
+- the skill reads well but is not being invoked reliably
+- the skill is firing on near-miss prompts
+- you have already finished at least a solid draft of the skill
+
+Do not optimize the description first if the skill body is still weak.
+Fix the capability before tuning the routing surface.
+
+## Step 1. Generate trigger eval queries
+
+Create roughly 20 realistic queries split between:
+
+- `should_trigger: true`
+- `should_trigger: false`
+
+Save them as JSON:
+
+```json
+[
+  {"query": "the user prompt", "should_trigger": true},
+  {"query": "another prompt", "should_trigger": false}
+]
+```
+
+The queries should feel like real user inputs, not toy examples.
+Include concrete details such as file names, work context, URLs, messy wording,
+typos, abbreviations, or ambiguous phrasing.
+
+Bad negative cases are obviously irrelevant.
+Good negative cases are near-misses that share vocabulary but should route
+somewhere else.
+
+## Step 2. Review the query set with the user
+
+Before running the loop, let the user review the query set.
+Use the local bundled HTML review asset in `assets/eval_review.html` to present
+and edit the eval set.
+
+The template workflow is:
+
+1. load the HTML template
+2. inject eval data, skill name, and current description
+3. write a temp HTML file
+4. open it for the user
+5. read the exported eval set from Downloads
+
+This step matters because poor trigger queries produce poor descriptions.
+
+## Step 3. Run the optimization loop
+
+Run the local bundled optimization loop from the skill root:
+
+```bash
+python scripts/run_loop.py \
+  --eval-set <path-to-trigger-eval.json> \
+  --skill-path <path-to-skill> \
+  --backend auto \
+  --max-iterations 5 \
+  --verbose
+```
+
+If the backend supports model selection, use the current host model when that
+keeps the results closer to the user's real experience.
+
+While the loop runs:
+
+- report progress to the user
+- watch train and held-out test behavior
+- avoid selecting descriptions based only on training performance
+
+## Step 4. Apply the result carefully
+
+Take the best description from the loop, update frontmatter, then show the user:
+
+- before
+- after
+- relevant scores or win rate
+
+Do not silently swap descriptions with no explanation.
+
+## Query Design Rules
+
+For positive cases:
+
+- vary phrasing from formal to casual
+- include cases where the user does not explicitly name the skill
+- include edge cases where this skill should win over a competing one
+
+For negative cases:
+
+- use adjacent domains and semantic overlaps
+- include ambiguous prompts a naive keyword match would misroute
+- avoid "obviously unrelated" filler
+
+## Host Notes
+
+On hosts that do not expose a shell-accessible backend for trigger evaluation,
+skip the automated loop and do a manual description review instead.
+
+On Codex-like judged backends, treat the results as a routing proxy rather than
+a perfect measurement of native host behavior.

package/skills/conversation-to-skill/references/evaluation-suite.md

@@ -0,0 +1,410 @@
+# Evaluation Suite
+
+Use this reference whenever `conversation-to-skill` decides a skill should be
+evaluated rather than merely drafted.
+
+This is the full evaluation suite.
+Do not reduce it to "run one prompt and eyeball it" unless the host truly
+cannot support more.
+
+This skill bundles the required evaluation support files locally:
+
+- `agents/grader.md`
+- `agents/comparator.md`
+- `agents/analyzer.md`
+- `eval-viewer/generate_review.py`
+- `assets/eval_review.html`
+- `scripts/*.py`
+- `references/compatibility.md`
+- `references/schemas.md`
+
+Prefer these local copies over reaching into another skill directory.
+
+## When To Use This
+
+Use the full suite when at least one of these is true:
+
+- the user asks to test, benchmark, compare, or improve a skill
+- the skill has objectively checkable outputs
+- the first draft is clearly weak and needs iteration
+- description quality or trigger accuracy matters
+
+You may skip the full suite when the user explicitly wants a draft only, or when
+the skill output is so subjective that formal grading would be fake precision.
+
+## Layout
+
+Choose the skill location first, then set up evaluation paths around it.
+
+- Global skill: `~/.agents/skills/<skill-name>`
+- Project-based skill: `<project-path>/.agents/skills/<skill-name>`
+- Eval workspace: sibling directory named `<skill-name>-workspace/`
+
+Within the workspace, organize by iteration:
+
+```text
+<skill-name>-workspace/
+├── skill-snapshot/              # optional baseline snapshot for existing skill
+├── iteration-1/
+│   ├── eval-0-<name>/
+│   ├── eval-1-<name>/
+│   └── benchmark.json
+└── iteration-2/
+```
+
+Within each eval directory, keep run outputs separated:
+
+```text
+eval-0-descriptive-name/
+├── eval_metadata.json
+├── with_skill/
+│   ├── outputs/
+│   ├── grading.json
+│   └── timing.json
+└── without_skill/              # or old_skill/
+    ├── outputs/
+    ├── grading.json
+    └── timing.json
+```
+
+Do not create every directory upfront.
+Create only what the current iteration needs.
+
+## Before Running Anything
+
+### 1. Write realistic test prompts
+
+Create 2-3 realistic prompts that a real user would plausibly type.
+Share them with the user when that feedback would help.
+
+Save them to `evals/evals.json`:
+
+```json
+{
+  "skill_name": "example-skill",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "User's task prompt",
+      "expected_output": "Description of expected result",
+      "files": []
+    }
+  ]
+}
+```
+
+Do not write assertions yet.
+You will draft them while the runs are in progress.
+
+### 2. Decide the baseline
+
+Use the right comparator:
+
+- New skill: `without_skill`
+- Existing skill being improved: snapshot the old version first, then compare against `old_skill`
+
+If improving an existing skill, snapshot before editing:
+
+```bash
+cp -r <skill-path> <workspace>/skill-snapshot/
+```
+
+### 3. Create eval metadata
+
+Each eval directory should contain an `eval_metadata.json`:
+
+```json
+{
+  "eval_id": 0,
+  "eval_name": "descriptive-name-here",
+  "prompt": "The user's task prompt",
+  "assertions": []
+}
+```
+
+Use descriptive eval names.
+Avoid generic names like `eval-0` when a better label exists.
+
+## Running The Suite
+
+This sequence is continuous.
+Do not stop after spawning runs or after creating the benchmark.
+
+### Step 1. Spawn all runs in the same turn
+
+For each test case, start both variants at once:
+
+- one run with the skill
+- one baseline run without the skill, or against the old snapshot
+
+Do not launch all with-skill runs first and defer baselines.
+Parallel launch keeps the comparison cleaner and faster.
+
+Use prompts shaped like:
+
+```text
+Execute this task:
+- Skill path: <path-to-skill>
+- Task: <eval prompt>
+- Input files: <eval files if any, or "none">
+- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
+- Outputs to save: <what the user actually cares about>
+```
+
+Baseline run:
+
+```text
+Execute this task:
+- Skill path: none                # or old snapshot path
+- Task: <eval prompt>
+- Input files: <eval files if any, or "none">
+- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/<baseline>/outputs/
+- Outputs to save: <same deliverables as with-skill>
+```
+
+### Step 2. Draft assertions while runs are executing
+
+Do not wait idly.
+While runs are in progress:
+
+- draft assertions for each eval
+- update `eval_metadata.json`
+- update `evals/evals.json`
+- explain to the user what each assertion checks if that context matters
+
+Good assertions are:
+
+- objective
+- descriptive
+- easy to understand in the benchmark viewer
+
+Bad assertions are vague or subjective.
+If quality depends on human judgment, keep that part qualitative.
+
+### Step 3. Capture timing as runs finish
+
+When each run completes, capture the timing data immediately:
+
+```json
+{
+  "total_tokens": 84852,
+  "duration_ms": 23332,
+  "total_duration_seconds": 23.3
+}
+```
+
+Save it to `timing.json` in the run directory.
+Do not delay this step if the host exposes timing only in completion
+notifications.
+
+### Step 4. Grade each run
+
+Evaluate each assertion against the outputs and save `grading.json`.
+
+The expectations array must use these exact fields:
+
+- `text`
+- `passed`
+- `evidence`
+
+Example:
+
+```json
+{
+  "expectations": [
+    {
+      "text": "Output includes a trigger-oriented description",
+      "passed": true,
+      "evidence": "Frontmatter description mentions both capability and trigger contexts."
+    }
+  ]
+}
+```
+
+If an assertion can be checked programmatically, prefer a script over manual
+inspection.
+
+### Step 5. Aggregate into a benchmark
+
+Once all runs are graded, aggregate the iteration into benchmark artifacts.
+
+Run the local aggregation script from the skill root:
+
+```bash
+python scripts/aggregate_benchmark.py <workspace>/iteration-N --skill-name <name>
+```
+
+Expected outputs:
+
+- `benchmark.json`
+- `benchmark.md`
+
+Put each `with_skill` result before its baseline counterpart in summaries so the
+comparison is easy to scan.
+
+### Step 6. Do an analyst pass
+
+Read the benchmark and look for patterns that averages hide:
+
+- assertions that always pass regardless of the skill
+- flaky or high-variance evals
+- speed or token regressions that are not buying quality
+- cases where the skill only improves one narrow prompt
+
+If a metric is non-discriminating, change the eval design rather than pretending
+it is useful.
+
+### Step 7. Generate the review viewer
+
+Do not stop at `benchmark.json`.
+Always generate a reviewable artifact for the human.
+
+Generate the review viewer with the local bundled script:
+
+```bash
+nohup python eval-viewer/generate_review.py \
+  <workspace>/iteration-N \
+  --skill-name "<name>" \
+  --benchmark <workspace>/iteration-N/benchmark.json \
+  > /dev/null 2>&1 &
+VIEWER_PID=$!
+```
+
+For iteration 2+, also pass:
+
+```bash
+--previous-workspace <workspace>/iteration-<N-1>
+```
+
+If a synthetic benchmark created no real outputs yet, add a minimal file such as
+`outputs/summary.md` so the viewer has something to render.
+
+### Step 8. Hand the viewer to the user in the same turn
+
+Do not make the user ask for the results viewer later.
+Tell them where it is immediately.
+
+If a server-backed viewer is available, say it is open and explain the two main
+tabs:
+
+- `Outputs`: prompt, output artifacts, grades, and feedback box
+- `Benchmark`: pass rate, time, token usage, and analyst observations
+
+### Step 9. Read feedback and iterate
+
+When the user finishes review, read `feedback.json`:
+
+```json
+{
+  "reviews": [
+    {"run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "..."},
+    {"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."}
+  ],
+  "status": "complete"
+}
+```
+
+Empty feedback usually means the user found that case acceptable.
+Focus changes on cases with specific complaints.
+
+If you started a viewer server, stop it afterwards:
+
+```bash
+kill $VIEWER_PID 2>/dev/null
+```
+
+## Improvement Loop
+
+Use the feedback to improve the skill, not to overfit it.
+
+Priorities:
+
+1. generalize from repeated failures instead of patching one exact prompt
+2. remove instructions that create busywork without quality gains
+3. explain why important behavior matters
+4. package repeated deterministic work into scripts when multiple runs rediscover it
+
+After revising the skill:
+
+1. rerun all evals into `iteration-<N+1>/`
+2. keep the same baseline logic unless there is a clear reason to change it
+3. regenerate the viewer with `--previous-workspace`
+4. collect user feedback again
+5. repeat until quality is acceptable or progress stalls
+
+Stop when:
+
+- the user is happy
+- feedback is empty across the board
+- the skill is no longer improving meaningfully
+
+## Blind Comparison
+
+If the user specifically wants a more rigorous A/B comparison between two skill
+versions, run a blind comparison:
+
+- give two outputs to an independent grader without saying which is which
+- have it judge quality
+- analyze why the winner won
+
+This is optional.
+Use it when normal human review is not enough.
+
+## Host-Specific Adaptation
+
+### Chat-only host
+
+If the host has no subagents:
+
+- run test cases one by one
+- skip baseline if independent comparison is impossible
+- present outputs directly in chat or save them for the user to inspect
+- focus on qualitative feedback
+- skip benchmarking if it would be fake rigor
+
+### Headless worker host
+
+If the host has no browser or display:
+
+- still run the full evaluation workflow
+- generate a static HTML review artifact instead of opening a live viewer
+- provide the exact output path to the user
+- expect feedback to arrive as a downloaded `feedback.json`
+
+For headless review generation, prefer:
+
+```bash
+python eval-viewer/generate_review.py \
+  <workspace>/iteration-N \
+  --skill-name "<name>" \
+  --benchmark <workspace>/iteration-N/benchmark.json \
+  --static <workspace>/iteration-N/review.html
+```
+
+## Packaging
+
+If the user wants a distributable skill package and the appropriate tooling is
+available, package it after the skill is stable.
+
+If the user wants packaging, use the local bundled script:
+
+```bash
+python scripts/package_skill.py <path/to/skill-folder>
+```
+
+## Final Rule
+
+If you chose evaluation, follow through.
+Do not stop after writing prompts.
+Do not stop after generating benchmarks.
+Do not stop after revising the skill once.
+
+The full suite is:
+
+- draft or revise the skill
+- run test cases
+- grade and benchmark them
+- generate a human-review artifact
+- collect feedback
+- improve the skill
+- rerun and compare again