@bgicli/bgicli 2.2.8 → 2.2.9

package/data/skills/hamelsmu-build-review-interface/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: build-review-interface
+description: >
+  Build a custom browser-based annotation interface tailored to your data for
+  reviewing LLM traces and collecting structured feedback. Use when you need to
+  build an annotation tool, review traces, or collect human labels.
+---
+
+# Build a Custom Annotation Interface
+
+## Overview
+
+Build an HTML page that loads traces from a data source (JSON/CSV file), displays one trace at a time with Pass/Fail buttons, a free-text notes field, and Next/Previous navigation. Save labels to a local file (CSV/SQLite/JSON). Then customize to the domain using the guidelines below.
+
+## Data Display
+
+Format all data in the most human-readable representation for the domain. Emails should look like emails. Code should have syntax highlighting. Markdown should be rendered. Tables should be tables. JSON should be pretty-printed and collapsible.
+
+- **Collapse repetitive elements.** If every trace shares the same system prompt, put it in a `<details>` toggle.
+- **Extract and surface key metadata.** If traces contain a property name, client type, or session ID buried in the data, extract it and display it prominently as a header or badge.
+- **Color-code by role or status.** Use left-border colors to distinguish user messages, assistant messages, tool calls, and system prompts at a glance.
+- **Group related elements visually.** Tool calls and their responses should be visually linked (indentation, shared border).
+- **Collapse what doesn't help judgment.** Verbose tool response JSON, intermediate reasoning steps, and debugging context go behind toggles.
+- **Highlight what matters most.** Make the primary content reviewers judge visually dominant. Bold key entities (prices, dates, names). Use font size and spacing to create hierarchy.
+- **Show the full trace.** Include all intermediate steps (tool calls, retrieved context, reasoning), not just the final output. Collapse them by default but keep them accessible.
+- **Sanitize rendered content.** Strip raw HTML from LLM outputs before rendering. Disable images in rendered markdown if they could be tracking pixels.
+
+## Feedback Collection
+
+Annotate at the trace level. The reviewer judges the whole trace, not individual spans.
+
+- Binary Pass/Fail buttons as the primary action.
+- Free-text notes field for the reviewer to describe what went wrong (or right).
+- Defer button for uncertain cases.
+- Auto-save on every action.
+
+Once you have established failure categories from error analysis, you can later add predefined failure mode tags as clickable checkboxes, dropdowns or picklists so reviewers can select from known categories in addition to writing notes.  But don't add these in the initial build.
+
+## Navigation and Status
+
+- Next/Previous buttons and keyboard arrow keys.
+- Trace counter showing position and progress ("12 of 87 remaining").
+- Jump to specific trace by ID.
+- Counts of labeled vs unlabeled traces.
+
+## Keyboard Shortcuts
+
+```
+Arrow keys = Navigate traces
+1 = Pass              2 = Fail
+D = Defer             U = Undo last action
+Cmd+S = Save          Cmd+Enter = Save and next
+```
+
+## Selecting Traces to Load
+
+Build the app to accept traces from any source (JSON/CSV file). Keep sampling logic outside the app in a separate script. Start with random sampling.
+
+## Additional Features
+
+**Reference panel:** Toggle-able panel showing ground truth, expected answers, or rubric definitions alongside the trace.
+
+**Filtering:** Filter traces by metadata dimensions relevant to the product (channel, user type, pipeline version).
+
+**Clustering:** Group traces by metadata or semantic similarity. Show representative traces per cluster with drill-down.
+
+## Design Checklist
+
+- [ ] Same layout, controls, and terminology on every trace
+- [ ] Pass and Fail buttons are visually distinct (color, size)
+- [ ] Keyboard shortcuts work for all primary actions
+- [ ] Full trace accessible even when sections are collapsed
+- [ ] Labels persist automatically without explicit save
+- [ ] Trace-level annotation (not span-level) as the default
+- [ ] All data rendered in its native format (markdown as HTML, code with highlighting, JSON pretty-printed, tables as HTML tables, URLs as clickable links)
+
+## Testing
+
+After building the interface, verify it with Playwright.
+
+**Visual review:** Take screenshots of the interface with representative trace data loaded. Review each screenshot for:
+- Layout and spacing: is the visual hierarchy clear? Can you immediately see what matters?
+- Readability: is all data rendered in its native format? Are there any raw JSON blobs, unrendered markdown, or unstyled content?
+- Aesthetics: does the interface look professional and clean? Would a domain expert use this?
+- Responsiveness: does the layout hold at different window sizes?
+
+**Functional test:** Write a Playwright script that performs a full annotation workflow:
+1. Load the app and verify traces are displayed
+2. Click Pass on a trace, verify the label is saved
+3. Click Fail on a trace, add a note, verify both are saved
+4. Click Defer, verify it is recorded
+5. Navigate forward and backward with buttons and keyboard shortcuts
+6. Verify the trace counter updates correctly
+7. Verify auto-save by reloading the page and checking labels persist
+8. Expand collapsed sections (system prompts, tool calls) and verify content is accessible
+9. Test that all keyboard shortcuts trigger the correct actions

package/data/skills/hamelsmu-error-analysis/SKILL.md

@@ -0,0 +1,164 @@
+---
+name: error-analysis
+description: >
+  Help the user systematically identify and categorize failure modes in an LLM
+  pipeline by reading traces. Use when starting a new eval project, after
+  significant pipeline changes (new features, model switches, prompt rewrites),
+  when production metrics drop, or after incidents.
+---
+
+# Error Analysis
+
+Guide the user through reading LLM pipeline traces and building a catalog of how the system fails.
+
+## Overview
+
+1. Collect ~100 representative traces
+2. Read each trace, judge pass/fail, and note what went wrong
+3. Group similar failures into categories
+4. Label every trace against those categories
+5. Compute failure rates to prioritize what to fix
+
+## Core Process
+
+### Step 1: Collect Traces
+
+Capture the full trace: input, all intermediate LLM calls, tool uses, retrieved documents, reasoning steps, and final output.
+
+**Target: ~100 traces.** This is roughly where new traces stop revealing new kinds of failures. The number depends on system complexity.
+
+**From real user data (preferred):**
+- Small volume: random sample
+- Large volume: sample across key dimensions (query type, user segment, feature area)
+- Use embedding clustering (K-means) to ensure diversity
+
+**From synthetic data (when real data is sparse):**
+- Use the generate-synthetic-data skill
+- Run synthetic queries through the full pipeline and capture complete traces
+
+### Step 2: Read Traces and Take Notes
+
+Present each trace to the user. For each one, ask: **did the system produce a good result?** Pass or Fail.
+
+For failures, note what went wrong. Focus on the **first thing that went wrong** in the trace — errors cascade, so downstream symptoms disappear when the root cause is fixed. Don't chase every issue in a single trace.
+
+Write observations, not explanations. "SQL missed the budget constraint" not "The model probably didn't understand the budget."
+
+**Template:**
+
+```
+| Trace ID | Trace | What went wrong | Pass/Fail |
+|----------|-------|-----------------|-----------|
+| 001      | [full trace] | Missing filter: pet-friendly requirement ignored in SQL | Fail |
+| 002      | [full trace] | Proposed unavailable times despite calendar conflicts | Fail |
+| 003      | [full trace] | Used casual tone for luxury client; wrong property type | Fail |
+| 004      | [full trace] | - | Pass |
+```
+
+**Heuristics:**
+- Do NOT start with a pre-defined failure list. Let categories emerge from what the user actually sees.
+- If the user is stuck articulating what feels wrong, prompt with common failure types: made-up facts, malformed output, ignored user requirements, wrong tone, tool misuse.
+
+### Step 3: Group Failures into Categories
+
+After reviewing 30-50 traces, start grouping similar notes into categories. Don't wait until all 100 are done — grouping early helps sharpen what to look for in the remaining traces. The categories will evolve. The goal is names that are specific and actionable, not perfect.
+
+1. Read through all the failure notes
+2. Group similar ones together
+3. Split notes that look alike but have different root causes
+4. Give each category a clear name and one-sentence definition
+
+**When to split vs. group:**
+
+Split these (different root causes):
+- "Made up property features (solar panels)" vs. "Made up client activity (scheduled a tour never requested)" — one fabricates external facts, the other fabricates user intent.
+
+Group these (same root cause):
+- "Missing bedroom count filter" + "Missing pet-friendly filter" + "Missing price range filter" → **Missing Query Constraints**
+
+**LLM-assisted clustering** (use only after the user has reviewed 30-50 traces):
+
+```
+Here are failure annotations from reviewing LLM pipeline traces.
+Group similar failures into 5-10 distinct categories.
+For each category, provide:
+- A clear name
+- A one-sentence definition
+- Which annotations belong to it
+
+Annotations:
+[paste annotations]
+```
+
+Always review LLM-suggested groupings with the user. LLMs cluster by surface similarity (e.g., grouping "app crashes" and "login is slow" because both mention login).
+
+**Aim for 5-10 categories** that are:
+- Distinct (each failure belongs to one category)
+- Clear enough that someone else could apply them consistently
+- Actionable (each points toward a specific fix)
+
+### Step 4: Label Every Trace
+
+Go back through all traces and apply binary labels (pass/fail) for each failure category. Each trace gets a column per category. Use whatever tool the user prefers — spreadsheet, annotation app (see build-review-interface), or a simple script.
+
+### Step 5: Compute Failure Rates
+
+```python
+failure_rates = labeled_df[failure_columns].sum() / len(labeled_df)
+failure_rates.sort_values(ascending=False)
+```
+
+The most frequent failure category is where to focus first.
+
+### Step 6: Decide What to Do About Each Failure
+
+Work through each category with the user in this order:
+
+**Can we just fix it?** Many failures have obvious fixes that don't need an evaluator at all:
+- The prompt never mentioned the requirement. Example: the LLM never includes photo links in emails because the prompt never asked for them. Add the instruction.
+- A tool is missing or misconfigured. Example: the user wants to reschedule but there's no rescheduling tool exposed to the LLM. Add the tool.
+- An engineering bug in retrieval, parsing, or integration. Fix the code.
+
+If a clear fix resolves the failure, do that first. Only consider an evaluator for failures that persist after fixing.
+
+**Is an evaluator worth the effort?** Not every remaining failure needs one. Building and maintaining evaluators has real cost. Ask the user:
+- Does this failure happen frequently enough to matter?
+- What's the business impact when it does happen? A rare failure that causes revenue loss may outrank a frequent failure that's merely annoying.
+- Will this evaluator actually get used to iterate on the system, or is it checkbox work?
+
+Reserve evaluators for failures the user will iterate on repeatedly. Start with the highest-frequency, highest-impact category.
+
+**For failures that warrant an evaluator:** prefer code-based checks (regex, parsing, schema validation) for anything objective. Use write-judge-prompt only for failures that require judgment. Critical requirements (safety, compliance) may warrant an evaluator even after fixing the prompt, as a guardrail.
+
+### Step 7: Iterate
+
+Expect 2-3 rounds of reviewing and refining categories. After each round:
+- Merge categories that overlap
+- Split categories that are too broad
+- Clarify definitions where the user would hesitate
+- Re-label traces with the refined categories
+
+## Stopping Criteria
+
+Stop reviewing when new traces aren't revealing new kinds of failures. Roughly: ~100 traces reviewed with no new failure types appearing in the last 20. The exact number depends on system complexity.
+
+## Trace Sampling Strategies
+
+When production volume is high, use a mix:
+
+| Strategy | When to Use | Method |
+|----------|------------|--------|
+| **Random** | Default starting point | Sample uniformly from recent traces |
+| **Outlier** | Surface unusual behavior | Sort by response length, latency, tool call count; review extremes |
+| **Failure-driven** | After guardrail violations or user complaints | Prioritize flagged traces |
+| **Uncertainty** | When automated judges exist | Focus on traces where judges disagree or have low confidence |
+| **Stratified** | Ensure coverage across user segments | Sample within each dimension |
+
+## Anti-Patterns
+
+- **Brainstorming failure categories before reading traces.** Read first, categorize what you find.
+- **Starting with pre-defined categories.** A fixed list causes confirmation bias. Let categories emerge.
+- **Skipping the user for initial review.** The user must review the first 30-50 traces to ground categories in domain knowledge.
+- **Using generic scores as categories.** "Hallucination score," "helpfulness score," "coherence score" are not grounded in the application's actual failure modes.
+- **Building evaluators before fixing obvious problems.** Fix prompt gaps, missing tools, and engineering bugs first.
+- **Treating this as a one-time activity.** Re-run after every significant change: new features, prompt rewrites, model switches, production incidents.

package/data/skills/hamelsmu-eval-audit/SKILL.md

@@ -0,0 +1,183 @@
+---
+name: eval-audit
+description: >
+  Audit an LLM eval pipeline and surface problems: missing error analysis,
+  unvalidated judges, vanity metrics, etc. Use when
+  inheriting an eval system, when unsure whether evals are trustworthy, or as a
+  starting point when no eval infrastructure exists. Do NOT use when the goal
+  is to build a new evaluator from scratch (use error-analysis,
+  write-judge-prompt, or validate-evaluator instead).
+---
+
+# Eval Audit
+
+Inspect an LLM eval pipeline and produce a prioritized list of problems with concrete next steps.
+
+## Overview
+
+1. Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards
+2. Run diagnostic checks across six areas
+3. Produce a findings report ordered by impact, with each finding linking to a fix
+
+## Prerequisites
+
+Access to eval artifacts (traces, evaluator configs, judge prompts, labeled data) via an observability MCP server or local files. If none exist, skip to "No Eval Infrastructure."
+
+## Connecting to Eval Infrastructure
+
+Check whether the user has an observability MCP server connected (Phoenix, Braintrust, LangSmith, Truesight or similar). If available, use it to pull traces, evaluator definitions, and experiment results. If not, ask for local files: CSVs, JSON trace exports, notebooks, or evaluation scripts.
+
+## Diagnostic Checks
+
+Work through each area below. Inspect available artifacts, determine whether the problem exists, and record a finding if it does.
+
+Prioritize findings by impact on the user's product. Present the most impactful findings first.
+
+### 1. Error Analysis
+
+**Check:** Has the user done systematic error analysis on real or synthetic traces?
+
+Look for: labeled trace datasets, failure category definitions, notes from trace review. If evaluators exist but no documented failure categories, error analysis was likely skipped.
+
+**Finding if missing:** Evaluators built without error analysis measure generic qualities ("helpfulness", "coherence") instead of actual failure modes. Start with `error-analysis`, or `generate-synthetic-data` first if no traces exist.
+
+See: [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/index.html), [LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/)
+
+**Check:** Were failure categories brainstormed or observed?
+
+Generic labels borrowed from research ("hallucination score", "toxicity", "coherence") suggest brainstorming. Application-grounded categories ("missing query constraints", "wrong client tone", "fabricated property features") suggest observation.
+
+**Finding if brainstormed:** Generic categories miss application-specific failures and produce evaluators that score well on paper but miss real problems. Re-do with `error-analysis`, starting from traces.
+
+See: [Who Validates the Validators?](https://arxiv.org/abs/2404.12272)
+
+### 2. Evaluator Design
+
+**Check:** Are evaluators binary pass/fail?
+
+Flag any that use Likert scales (1-5), letter grades (A-F), or numeric scores without a clear pass/fail threshold.
+
+**Finding if not binary:** Likert scales are difficult to calibrate. Annotators disagree on the difference between a 3 and a 4, and judges inherit that noise. Consider converting to binary pass/fail with explicit definitions using `write-judge-prompt`.
+
+See: [Creating an LLM Judge That Drives Business Results](https://hamel.dev/blog/posts/llm-judge/)
+
+**Check:** Do LLM judge prompts target specific failure modes?
+
+Flag any that evaluate holistically ("Is this response helpful?", "Rate the quality of this output").
+
+**Finding if vague:** Holistic judges produce unactionable verdicts. Each judge should check exactly one failure mode with explicit pass/fail definitions and few-shot examples. Use `write-judge-prompt`.
+
+**Check:** Are code-based checks used where possible?
+
+Flag LLM judges used for objectively checkable criteria: format validation, constraint satisfaction, keyword presence, schema conformance.
+
+**Finding if over-relying on judges:** Replace objective checks with code (regex, parsing, schema validation, execution tests). Reserve LLM judges for criteria requiring interpretation.
+
+**Check:** Are similarity metrics used as primary evaluation?
+
+Flag ROUGE, BERTScore, cosine similarity, or embedding distance used as the main evaluator for generation quality.
+
+**Finding if present:** These metrics measure surface-level overlap, not correctness. They suit retrieval ranking but not generation evaluation. Replace with binary evaluators grounded in specific failure modes.
+
+See: [LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/)
+
+### 3. Judge Validation
+
+**Check:** Are LLM judges validated against human labels?
+
+Look for: confusion matrices, TPR/TNR measurements, alignment scores. Judges in production with no validation data is a critical finding.
+
+**Finding if unvalidated:** An unvalidated judge may consistently miss failures or flag passing traces. Measure alignment using TPR and TNR on a held-out test set. Use `validate-evaluator`.
+
+See: [Creating an LLM Judge That Drives Business Results](https://hamel.dev/blog/posts/llm-judge/)
+
+**Check:** Is alignment measured with TPR/TNR or with raw accuracy?
+
+Flag "accuracy", "percent agreement", or Cohen's Kappa as the primary alignment metric.
+
+**Finding if using accuracy:** With class imbalance, raw accuracy is misleading: a judge that always says "Pass" gets 90% accuracy when 90% of traces pass but catches zero failures. Use TPR and TNR, which map directly to bias correction. Use `validate-evaluator`.
+
+**Check:** Is there a proper train/dev/test split?
+
+Check whether few-shot examples in judge prompts come from the same data used to measure judge performance.
+
+**Finding if leaking:** Using evaluation data as few-shot examples inflates alignment scores and hides real judge failures. Split into train (few-shot source), dev (iteration), and test (final measurement). Use `validate-evaluator`.
+
+### 4. Human Review Process
+
+**Check:** Who is reviewing traces?
+
+Determine whether domain experts or outsourced annotators are labeling data.
+
+**Finding if outsourced without domain expertise:** General annotators catch formatting errors but miss domain-specific failures (wrong medical dosage, incorrect legal citation, mismatched property features). Involve a domain expert.
+
+See: [A Field Guide to Improving AI Products](https://hamel.dev/blog/posts/field-guide/)
+
+**Check:** Are reviewers seeing full traces or just final outputs?
+
+**Finding if output-only:** Reviewing only the final output hides where the pipeline broke. Show the full trace: input, intermediate steps, tool calls, retrieved context, and final output.
+
+**Check:** How is data displayed to reviewers?
+
+Flag raw JSON, unformatted text, or spreadsheets with trace data in cells.
+
+**Finding if raw format:** Reviewers spend effort parsing data instead of judging quality. Format in natural representation: render markdown, syntax-highlight code, display tables as tables. Use `build-review-interface`.
+
+See: [LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/)
+
+### 5. Labeled Data
+
+**Check:** Is there enough labeled data?
+
+For error analysis, ~100 traces is the rough target for saturation. For judge validation, ~50 Pass and ~50 Fail examples are needed for reliable TPR/TNR. If labeled data is sparse, collect more by sampling traces more effectively:
+
+- **Random:** Always include a random sample alongside other strategies to discover unknown issues.
+- **Clustering:** Group traces by semantic similarity and review representatives from each cluster.
+- **Data analysis:** Analyze statistics on latency, turns, tool calls, and tokens for outliers.
+- **Classification:** Use existing evals, a predictive model, or an LLM to surface problematic traces. Use with caution.
+- **Feedback:** Use explicit customer feedback (complaints, thumbs-down signals) to filter traces.
+
+**Finding if insufficient:** Small datasets produce unreliable failure rates and wide confidence intervals. Use the sampling strategies above to collect more labeled data, or supplement with `generate-synthetic-data`.
+
+### 6. Pipeline Hygiene
+
+**Check:** Is error analysis re-run after significant changes?
+
+Check when error analysis was last performed relative to model switches, prompt rewrites, new features, or production incidents.
+
+**Finding if stale:** Failure modes shift after pipeline changes, and evaluators built for the old pipeline miss new failure types. Re-run error analysis after every significant change.
+
+**Check:** Are evaluators maintained?
+
+Look for periodic re-validation of judges or refreshed evaluation datasets.
+
+**Finding if set-and-forget:** Evaluators degrade as the pipeline evolves. Re-validate judges against fresh human labels and update eval datasets to reflect current usage.
+
+## No Eval Infrastructure
+
+If the user has no eval artifacts (no traces, no evaluators, no labeled data):
+
+1. Start with `error-analysis` on a sample of real traces.
+2. If no production data exists, use `generate-synthetic-data` to create test inputs, run them through the pipeline, then apply `error-analysis` to the resulting traces.
+3. Do not recommend building evaluators, judges, or dashboards before completing error analysis.
+
+## Report Format
+
+Present findings ordered by impact. For each:
+
+```
+### [Problem Title]
+**Status:** [Problem exists / OK / Cannot determine]
+[1-2 sentence explanation of the specific problem found]
+**Fix:** [Concrete action, referencing a skill or article]
+```
+
+Group under the six diagnostic areas. Omit areas where no problems were found.
+
+## Anti-Patterns
+
+- Running the audit as a checklist without inspecting actual artifacts.
+- Reporting generic advice disconnected from what was found in the user's pipeline.
+- Recommending evaluators before error analysis is complete.
+- Suggesting LLM judges for failures that code-based checks can handle.
+- Treating this audit as a one-time event. Re-audit after significant pipeline changes.

package/data/skills/hamelsmu-evaluate-rag/SKILL.md

@@ -0,0 +1,177 @@
+---
+name: evaluate-rag
+description: >
+  Guides evaluation of RAG pipeline retrieval and generation quality.
+  Use when evaluating a retrieval-augmented generation system, measuring retrieval quality,
+  assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval
+  testing, or optimizing chunking strategies.
+---
+
+# Evaluate RAG
+
+## Overview
+
+1. Do error analysis on end-to-end traces first. Determine whether failures come from retrieval, generation, or both.
+2. Build a retrieval evaluation dataset: queries paired with relevant document chunks.
+3. Measure retrieval quality with Recall@k (most important for first-pass retrieval).
+4. Evaluate generation separately: faithfulness (grounded in context?) and relevance (answers the query?).
+5. If retrieval is the bottleneck, optimize chunking via grid search before tuning generation.
+
+## Prerequisites
+
+Complete error analysis on RAG pipeline traces before selecting metrics. Inspect what was retrieved vs. what the model needed. Determine whether the problem is retrieval, generation, or both. Fix retrieval first.
+
+## Core Instructions
+
+### Evaluate Retrieval and Generation Separately
+
+Measure each component independently. Use the appropriate metric for each retrieval stage:
+
+- **First-pass retrieval:** Optimize for Recall@k. Include all relevant documents, even at the cost of noise.
+- **Reranking:** Optimize for Precision@k, MRR, or NDCG@k. Rank the most relevant documents first.
+
+### Building a Retrieval Evaluation Dataset
+
+You need queries paired with ground-truth relevant document chunks.
+
+**Manual curation (highest quality):** Write realistic questions and map each to the exact chunk(s) containing the answer.
+
+**Synthetic QA generation (scalable):** For each document chunk, prompt an LLM to extract a fact and generate a question answerable only from that fact.
+
+Synthetic QA prompt template:
+
+```
+Given a chunk of text, extract a specific, self-contained fact from it.
+Then write a question that is directly and unambiguously answered
+by that fact alone.
+
+Return output in JSON format:
+{ "fact": "...", "question": "..." }
+
+Chunk: "{text_chunk}"
+```
+
+**Adversarial question generation:** Create harder queries that resemble content in multiple chunks but are only answered by one.
+
+Process:
+1. Select target chunk A containing a clear fact.
+2. Find similar chunks B, C using embedding search (chunks that share terminology but lack the answer).
+3. Prompt the LLM to write a question using terminology from B and C that only chunk A answers.
+
+Example:
+- Chunk A: "In April 2020, the company reported a 17% drop in quarterly revenue, its largest decline since 2008."
+- Chunk B: "The company experienced significant losses in 2008 during the financial crisis."
+- Generated question: "When did the company experience its largest revenue decline since the 2008 financial crisis?"
+
+Only chunk A contains the answer. Chunk B is a plausible distractor.
+
+**Filtering synthetic questions:** Rate synthetic queries for realism using few-shot LLM scoring. Keep only those rated realistic (4-5 on a 1-5 scale). Likert scoring is appropriate here, since the goal is fuzzy ranking for dataset curation, not measuring failure rates.
+
+### Retrieval Metrics
+
+**Recall@k:** Fraction of relevant documents found in the top k results.
+
+```
+Recall@k = (relevant docs in top k) / (total relevant docs for query)
+```
+
+Prioritize recall for first-pass retrieval. LLMs can ignore irrelevant content but cannot generate from missing content.
+
+**Precision@k:** Fraction of top k results that are relevant.
+
+```
+Precision@k = (relevant docs in top k) / k
+```
+
+Use for reranking evaluation.
+
+**Mean Reciprocal Rank (MRR):** How early the first relevant document appears.
+
+```
+MRR = (1/N) * sum(1/rank_of_first_relevant_doc)
+```
+
+Best for single-fact lookups where only one key chunk is needed.
+
+**NDCG@k (Normalized Discounted Cumulative Gain):** For graded relevance where documents have varying utility. Rewards placing more relevant items higher.
+
+```
+DCG@k  = sum over i=1..k of: rel_i / log2(i+1)
+IDCG@k = DCG@k with documents sorted by decreasing relevance
+NDCG@k = DCG@k / IDCG@k
+```
+
+Caveat: Optimal ranking of weakly relevant documents can outscore a highly relevant document ranked lower. Supplement with Recall@k.
+
+**Choosing k:** k varies by query type. A factual lookup uses k=1-2. A synthesis query ("summarize market trends") uses k=5-10.
+
+#### Metric Selection
+
+| Query Type | Primary Metric |
+|---|---|
+| Single-fact lookups | MRR |
+| Broad coverage needed | Recall@k |
+| Ranked quality matters | NDCG@k or Precision@k |
+| Multi-hop reasoning | Two-hop Recall@k |
+
+### Evaluating and Optimizing Chunking
+
+Treat chunking as a tunable hyperparameter. Even with the same retriever, metrics vary based on chunking alone.
+
+**Grid search for fixed-size chunking:** Test combinations of chunk size and overlap. Re-index the corpus for each configuration. Measure retrieval metrics on your evaluation dataset.
+
+Example search grid:
+
+| Chunk size | Overlap | Recall@5 | NDCG@5 |
+|-----------|---------|----------|--------|
+| 128 tokens | 0 | 0.82 | 0.69 |
+| 128 tokens | 64 | 0.88 | 0.75 |
+| 256 tokens | 0 | 0.86 | 0.74 |
+| 256 tokens | 128 | 0.89 | 0.77 |
+| 512 tokens | 0 | 0.80 | 0.72 |
+| 512 tokens | 256 | 0.83 | 0.74 |
+
+**Content-aware chunking:** When fixed-size chunks split related information:
+- Use natural document boundaries (sections, paragraphs, steps).
+- Augment chunks with context: prepend document title and section headings to each chunk before embedding.
+
+### Evaluating Generation Quality
+
+After confirming retrieval works, evaluate what the LLM does with the retrieved context along two dimensions:
+
+**Answer faithfulness:** Does the output accurately reflect the retrieved context? Check for:
+- **Hallucinations:** Information absent from source documents. In RAG, even correct facts from the LLM's own knowledge count as hallucinations.
+- **Omissions:** Relevant information from the context ignored in the output.
+- **Misinterpretations:** Context information represented inaccurately.
+
+**Answer relevance:** Does the output address the original query? An answer can be faithful to the context but fail to answer what the user asked.
+
+Use error analysis to discover specific manifestations in your pipeline. Identify what kind of information gets hallucinated and which constraints get omitted.
+
+#### Diagnosing Failures by Metric Pattern
+
+| Context Relevance | Faithfulness | Answer Relevance | Diagnosis |
+|---|---|---|---|
+| High | High | Low | Generator attended to wrong section of a correct document |
+| High | Low | -- | Hallucination or misinterpretation of retrieved content |
+| Low | -- | -- | Retrieval problem. Fix chunking, embeddings, or query preprocessing |
+
+### Multi-Hop Retrieval Evaluation
+
+For queries requiring information from multiple chunks:
+
+**Two-hop Recall@k:** Fraction of 2-hop queries where both ground-truth chunks appear in the top k results.
+
+```
+TwoHopRecall@k = (1/N) * sum(1 if {Chunk1, Chunk2} ⊆ top_k_results)
+```
+
+Diagnose failures by classifying: hop 1 miss, hop 2 miss, or rank-out-of-top-k.
+
+## Anti-Patterns
+
+- Using a single end-to-end correctness metric without separating retrieval and generation measurement.
+- Jumping directly to metrics without reading traces first.
+- Overfitting to synthetic evaluation data. Validate against real user queries regularly.
+- Using similarity metrics (ROUGE, BERTScore, cosine similarity) as primary generation evaluation. Use binary evaluators driven by error analysis.
+- Evaluating generation without checking context grounding.