quiver-cli 1.3.1 → 2.0.0

package/template/.agents/skills/integrations/langfuse/references/judge-calibration.md

@@ -1,293 +0,0 @@
----
-name: langfuse-judge-calibration
-description: Calibrate and validate LLM-as-a-Judge evaluators against dataset ground truth. Runs the judge prompt as a Langfuse dataset experiment, compares judge outputs with dataset item expected outputs, and reports simple accuracy or advanced confusion-matrix metrics. Use this guide whenever a user asks if their LLM judge is actually useful, aligned with human judgment, or safe to trust for monitoring decisions.
-metadata:
-  required_access:
-    - CODEBASE
-    - LANGFUSE_PROJECT_SCRIPT
----
-
-# Judge Calibration (LLM-as-a-Judge)
-
-## Goal
-
-Validate judge outputs against human labels using the smallest reliable workflow
-for the user's goal.
-
-Default to a **Langfuse dataset experiment** when the user has a Langfuse
-dataset or wants results in the Langfuse Experiments UI.
-
-Default to **simple calibration** unless the user asks for deeper metrics,
-split-based validation, thresholding, or production automation.
-
-## 1) Choose the calibration mode
-
-### Simple calibration
-
-Use this when the user wants a quick answer like "does this judge basically
-match human labels?" or explicitly asks for accuracy only.
-
-- No train/dev/test split is required.
-- Compute `exact_match` for each valid row.
-- Report valid sample size, invalid-label count, accuracy, and a short
-  recommendation.
-- Do not include Precision/Recall/F1, TPR/TNR, denominator notes, or top failure
-  direction unless the user asks for advanced metrics.
-
-### Advanced calibration
-
-Use this when the user asks for confusion matrix metrics, thresholds,
-production monitoring, high-stakes automation, or train/test-style validation.
-
-- If split labels exist, keep train/dev/test separate.
-- If no split exists, compute metrics on the provided rows and state that this is
-  not a held-out final quality claim.
-- Compute TP/FP/FN/TN and derived metrics.
-- Use `references/error-analysis.md` for qualitative diagnosis of disagreements.
-
-## 2) Primary workflow
-
-1. Confirm the dataset name, ground-truth label location in `expectedOutput`,
-   judge prompt name/version, judge model, and label vocabulary.
-2. Choose simple or advanced mode. If ambiguous, use simple mode.
-3. Run the judge prompt against each dataset item input as a Langfuse experiment.
-4. Compare the judge output to `item.expected_output` in evaluator functions.
-5. Return the matching report format from section 7.
-
-## 3) Langfuse experiment workflow
-
-Use the SDK experiment runner as the default implementation. A Langfuse-hosted
-dataset automatically creates a dataset run that can be inspected and compared
-in the Langfuse UI.
-
-Before implementing, you **must** retrieve the current experiment SDK
-documentation from the Langfuse docs — do not rely on memory, the SDK changes
-frequently. Fetch these pages (see SKILL.md section 2 for retrieval methods):
-
-- [Experiments via SDK](https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk) — primary reference for `dataset.run_experiment`, task/evaluator signatures, and `Evaluation` return shape
-- [Datasets](https://langfuse.com/docs/evaluation/experiments/datasets) — dataset item structure (`input`, `expectedOutput`) and how to load a hosted dataset
-- [Experiments data model](https://langfuse.com/docs/evaluation/experiments/data-model) — how runs, items, and scores relate in the UI
-
-High-level shape of a simple-mode calibration experiment:
-
-```
-load dataset and judge prompt from Langfuse
-define POSITIVE / NEGATIVE label set
-
-task(item):
-    compile judge prompt with item.input
-    call judge model
-    return normalized label
-    # never read item.expected_output here — that would leak the answer
-
-item_evaluator(output, expected_output):
-    if either label is outside the allowed set:
-        return invalid (excluded from accuracy denominator)
-    return exact_match score (0 or 1)
-
-run_evaluator(item_results):
-    accuracy = matches / valid_rows
-    return aggregate score + invalid_label count
-
-dataset.run_experiment(task, [item_evaluator], [run_evaluator],
-                      metadata={calibration_mode, judge_prompt, labels})
-```
-
-For advanced calibration, the item evaluator emits one score per confusion-matrix
-cell (`judge-is-tp`, `judge-is-fp`, `judge-is-fn`, `judge-is-tn`), and the run
-evaluator aggregates them into Precision / Recall / F1 / TPR / TNR. See section
-6 for the metric definitions and zero-denominator guardrails.
-
-Rules:
-- Use a Langfuse-hosted dataset when the user wants a real Langfuse experiment.
-  Local SDK datasets create traces and scores, but not Langfuse dataset runs.
-- The dataset item `input` must contain everything needed to run the judge
-  prompt. The dataset item `expectedOutput` must contain the ground-truth label.
-- Never pass `expectedOutput` into the judge prompt or task. That would leak the
-  answer and invalidate calibration.
-- Return `Evaluation(...)` objects from item and run evaluators for stable SDK
-  formatting and score ingestion.
-- Store prompt name/version, judge model, label vocabulary, dataset version, and
-  calibration mode in run metadata.
-- Use a unique run name so the experiment appears as a separate dataset run.
-
-## 4) Label validation
-
-Never silently treat unknown labels as negative.
-
-- For binary judges, define the positive and negative labels before computing
-  confusion-matrix metrics.
-- Normalize only deterministic differences such as surrounding whitespace or
-  casing, and mention that normalization was applied.
-- Rows where `expected` or `actual` is outside the allowed label set are invalid.
-  Exclude them from metric denominators and report the invalid count.
-- For multi-class judges, use simple `exact_match` / accuracy unless the user
-  defines one positive class for binary metrics.
-
-Example binary labels:
-- Positive: `ESCALATE`
-- Negative: `RESOLVE`
-
-## 5) Simple metrics
-
-For each valid row:
-
-- `exact_match = 1 if actual == expected else 0`
-
-Aggregate:
-
-- `accuracy = sum(exact_match) / valid_rows`
-
-If `valid_rows == 0`, report that accuracy is undefined and ask for valid
-expected/actual labels.
-
-## 6) Advanced metrics
-
-### Dataset and split discipline
-
-Use split discipline only when the user asks for advanced validation or final
-quality claims.
-
-- **Train**: optional few-shot examples for the judge prompt
-- **Dev**: iterative prompt/model refinement
-- **Test**: single final calibration pass
-
-Do not tune on the same rows used to claim final quality. Use balanced classes
-in dev/test when possible so both error directions are measurable.
-
-### Per-row classification mapping
-
-For each row with `expected` and `actual` labels:
-
-- **TP**: expected = positive and actual = positive
-- **FP**: expected = negative and actual = positive
-- **FN**: expected = positive and actual = negative
-- **TN**: expected = negative and actual = negative
-
-Also compute:
-- `exact_match = 1 if actual == expected else 0`
-
-### Aggregate metrics
-
-From aggregate counts:
-- `accuracy = (TP + TN) / valid_rows`
-- `precision = TP / (TP + FP)`
-- `recall = TP / (TP + FN)`
-- `f1 = 2 * precision * recall / (precision + recall)`
-- `TPR = TP / (TP + FN)`
-- `TNR = TN / (TN + FP)`
-
-Guardrails:
-- if `TP + FP == 0`, precision is undefined (report null + note)
-- if `TP + FN == 0`, recall and TPR are undefined (report null + note)
-- if `TN + FP == 0`, TNR is undefined (report null + note)
-- if `precision + recall == 0`, set `f1 = 0`
-
-### Advanced quality gates
-
-Before trusting the judge on production traffic:
-
-1. **Split integrity**: no leakage from held-out rows into prompt examples.
-2. **Confusion matrix sanity**: `TP + FP + FN + TN == valid_rows`.
-3. **Metric recomputation check**: recompute aggregate stats from row-level
-   flags and compare.
-4. **TPR/TNR review**: inspect both directions for class-direction bias.
-5. **Acceptance criteria**: before the held-out test, agree on thresholds for
-   the relevant metrics based on the costs of false positives and false
-   negatives and whether the judge informs monitoring or automation. Do not
-   infer "ship" from a universal cutoff.
-
-## 7) Report format
-
-### Simple report
-
-Return only:
-- dataset name and dataset run URL when available
-- valid rows / total rows
-- invalid-label count
-- accuracy
-- one-sentence recommendation
-
-### Advanced report
-
-Add:
-- confusion matrix: TP, FP, FN, TN
-- accuracy, precision, recall, F1, TPR, TNR
-- denominator notes for undefined metrics
-- top failure direction: false positives or false negatives
-- recommendation: ship, iterate, collect more labels, or do not automate
-
-## 8) Langfuse implementation notes
-
-Prefer SDK experiment evaluators for score creation. They attach item-level
-scores to the experiment traces and run-level scores to the dataset run.
-
-Use manual REST score creation only as a fallback when not using the SDK
-experiment runner, or for local smoke tests. See
-[Scores via SDK](https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk)
-and the [Scores API reference](https://langfuse.com/docs/api) for the current
-payload shape. Before using the CLI for score creation, inspect its current
-schema and action help instead of assuming resource names, arguments, or
-capabilities from a known package version.
-
-Score names to emit:
-
-Simple mode:
-- `judge-exact-match`
-- `judge-accuracy`
-
-Advanced mode:
-- `judge-exact-match`
-- `judge-is-tp`
-- `judge-is-fp`
-- `judge-is-fn`
-- `judge-is-tn`
-
-Recommended metadata:
-- `expected_label`, `actual_label`
-- calibration mode: `simple` or `advanced`
-- positive/negative labels when binary metrics are used
-- evaluator prompt name+version
-- dataset/split version when used
-- run identifier
-
-## 9) Classification logic (pseudo-code)
-
-The per-row classification each evaluator must perform:
-
-```
-normalize(label) = strip whitespace, uppercase
-ALLOWED = {POSITIVE, NEGATIVE}
-
-classify(expected, actual):
-    expected, actual = normalize(expected), normalize(actual)
-
-    if expected ∉ ALLOWED or actual ∉ ALLOWED:
-        mark row invalid → exclude from denominators
-
-    exact_match = (expected == actual)
-    is_tp = (expected == POSITIVE and actual == POSITIVE)
-    is_fp = (expected == NEGATIVE and actual == POSITIVE)
-    is_fn = (expected == POSITIVE and actual == NEGATIVE)
-    is_tn = (expected == NEGATIVE and actual == NEGATIVE)
-```
-
-## 10) Common failure modes
-
-- label vocabulary not constrained (judge outputs free text instead of strict
-  labels)
-- positive/negative label inversion between annotators and evaluator code
-- leaking `expectedOutput` into the judge task instead of only the evaluator
-- using local SDK data when the user expects a Langfuse dataset run in the UI
-- reporting only accuracy when classes are imbalanced and error direction matters
-- calculating F1 without explicit zero-denominator handling
-- using advanced validation claims without a held-out split
-
-## 11) What to do after calibration
-
-- If simple accuracy is enough: report it and stop.
-- If metrics are weak and advanced validation is needed: iterate prompt and
-  few-shots on dev data only.
-- If metrics pass: freeze the baseline and monitor drift over time.
-- For qualitative diagnosis of disagreements, switch to
-  `references/error-analysis.md`.

package/template/.agents/skills/integrations/langfuse/references/prompt-engineering.md

@@ -1,35 +0,0 @@
----
-name: langfuse-prompt-engineering
-description: Write or change prompts in Langfuse or code. Use whenever the user asks to create, edit, rewrite, debug, tune, or otherwise modify a prompt, including a small wording or instruction change. Distinct from prompt-migration and judge-calibration.
-metadata:
-  required_access: []
----
-
-# Prompt Engineering
-
-## Universal principles
-
-1. **Be specific.** State the required output format, constraints, and, when order matters, numbered steps. Write for someone with minimal context.
-2. **Use labeled sections** for the role, instructions, examples, and context. Use XML tags when they help distinguish rules, data, and variables.
-3. **Give the model a role.** A focused system-prompt sentence can steer tone and behavior.
-4. **Explain why important rules exist** so the model can apply them to adjacent cases.
-
-## Adjusting an existing prompt
-
-1. **Pin down the failure first.** Use concrete misbehaving outputs and name the exact failure mode: wrong format, ignored instruction, wrong tone, hallucination, or another observable error. The task and data show what could fail, not what did fail; do not write a fix before observing the failure.
-2. **Trace the failure to the prompt.** Look for a missing, ambiguous, or conflicting instruction.
-3. **Fix the failure class, not the example.** Generalize the observed mistake into an error class and write the instruction at that level, in your own words. Cover adjacent cases, but do not add definitions, rules, or examples for unobserved problems or encode the failing example itself. If a generalized instruction does not work, report the failure instead of narrowing the rule to that case.
-4. **Preserve existing behavior.** Before changing text, identify what behavior it protects. Avoid contradicting other rules or breaking cases that already work.
-5. **Make the smallest testable edit.** Change one cause per attempt so you can attribute the result. Avoid extra length, emphasis, or all-caps. In Langfuse, save each attempt as a new version or label so you can compare and roll back.
-
-## Model-specific tuning
-
-Identify the target model if you can — tuning differs by model, so when you know which model the prompt runs on, follow that model's own prompting guidance.
-
-| Technique | Standard (GPT) | Reasoning (Claude latest) |
-|-----------|----------------|---------------------------|
-| Instruction density | High, prescriptive | Goal-level; don't micro-manage steps |
-| Reasoning | Add explicit chain-of-thought | Native thinking; "think thoroughly" + tune `effort` |
-| Emphasis language | Strong directives fine | Dial back `MUST`/`CRITICAL` |
-| Output format | Prefilling/scaffolds | Prefill removed (4.6+); structured outputs + "no preamble" |
-| Roles | `developer` > `user` > `assistant` | `system` role + `user`/`assistant` |

package/template/.agents/skills/integrations/langfuse/references/prompt-migration.md

@@ -1,79 +0,0 @@
----
-name: langfuse-prompt-migration
-description: Migrate hardcoded prompts to Langfuse for version control and deployment-free iteration. Use when user wants to externalize prompts, move prompts to Langfuse, or set up prompt management.
-metadata:
-  required_access:
-    - CODEBASE
-    - LANGFUSE_PROJECT_SCRIPT
----
-
-# Langfuse Prompt Migration
-
-Migrate hardcoded prompts into Langfuse-managed prompts. The API mechanics (`create_prompt`, `get_prompt`, `.compile()`, linking to traces) are in the docs — fetch them at execution time.
-
-## Prerequisites
-
-Verify credentials exist — check presence only, never print the secret key (its value would land in the agent's context and transcripts):
-
-```bash
-[ -n "$LANGFUSE_PUBLIC_KEY" ] && echo "public key: set" || echo "public key: missing"
-[ -n "$LANGFUSE_SECRET_KEY" ] && echo "secret key: set" || echo "secret key: missing"
-[ -n "${LANGFUSE_BASE_URL:-${LANGFUSE_HOST:-}}" ] && echo "base url: set" || echo "base url: missing"
-```
-
-Use `LANGFUSE_BASE_URL` for current SDKs. If only `LANGFUSE_HOST` is set, export `LANGFUSE_BASE_URL="$LANGFUSE_HOST"`. If a CLI expects `LANGFUSE_HOST` and only `LANGFUSE_BASE_URL` is set, export `LANGFUSE_HOST="$LANGFUSE_BASE_URL"`.
-
-If credentials or the base URL are missing, ask the user to set them in their shell or a `.env` file. Do not ask them to paste secret keys into chat.
-
-## 1. Inventory every prompt (before writing any code)
-
-For each prompt, record:
-
-- **Name**: lowercase, hyphenated (e.g. `chat-assistant`)
-- **Source file**: where the prompt text lives
-- **Code file to refactor**: the file that USES the prompt. For asset files (`.txt`/`.yaml`/`.md`), this is the file that loads the asset, not the asset itself
-- **Type**: `chat` (message array) or `text` (plain string)
-- **Variables**: values interpolated in, converted to `{{var}}`
-- **Content**: the actual text to upload
-
-Before choosing `text` or `chat`, fetch and follow [Chat vs Text Prompts](https://langfuse.com/docs/prompt-management/data-model#text-vs-chat-prompts). Do not default unrelated prompt flows to one prompt type.
-
-Prompts typically live in OpenAI message arrays, Anthropic system arguments, LangChain prompt templates, Vercel AI system/prompt fields, and raw multi-line strings near LLM calls.
-
-## 2. Convert templating and decide structure
-
-**Variable syntax:** Langfuse substitutes only double-brace `{{var}}`. Convert every single-brace form during upload — `{var}`, `${var}`, f-string `{var}`, `.format(var=...)`, and string concatenation all become `{{var}}`. Uploading `{var}` will silently fail to substitute.
-
-**Complex templates:** Langfuse has no conditionals, loops, or filters. If the code uses them (e.g. Jinja `{% if %}`/`{% for %}`), either pre-compute the value in code and pass a plain `{{variable}}` (recommended), or store the raw template and compile client-side — which loses Playground preview and UI experiments. See https://langfuse.com/docs/prompt-management/features/variables and the [external templating FAQ](https://langfuse.com/faq/all/using-external-templating-libraries).
-
-**What to make a variable vs. keep hardcoded:**
-
-| Make variable | Keep hardcoded |
-|---------------|----------------|
-| User-specific (`{{user_name}}`) | Output format instructions |
-| Dynamic content (`{{context}}`) | Safety guardrails |
-| Per-request (`{{query}}`) | Persona / personality |
-| Environment-specific (`{{company_name}}`) | Static examples |
-
-**Naming:** lowercase-hyphenated, feature-based (`document-summarizer`), hierarchical for related prompts (`support/triage`), prefix subprompts with `_` (`_base-personality`).
-
-Before extracting subprompts, fetch and follow [Prompt Composability](https://langfuse.com/docs/prompt-management/features/composability). Use composition for the reuse and shared-maintenance cases described there, not merely to decompose one coherent prompt flow.
-
-## 3. Present the plan, then create and refactor
-
-When the user explicitly requests a migration and credentials work, treat that request as authorization to create prompts and refactor the call sites. Present the inventory and plan as a progress update, then continue.
-
-Ask only when a materially different design choice would change behavior, required credentials are unavailable, or destructive cleanup needs approval. Do not stop merely to confirm names, prompt types, or optional tracing.
-
-Then:
-
-- Create the prompts (label migrated prompts `production` — they're already live) and refactor call sites to fetch each prompt from Langfuse and compile its variables in. The SDK calls differ across Python and JS/TS — fetch the current docs: https://langfuse.com/docs/prompt-management/get-started
-- Fetch by the `production` label
-- If the codebase already has Langfuse tracing (decorators, an instrumented client, or manual spans), link prompts so you can see which version produced each response. See https://langfuse.com/docs/prompt-management/features/link-to-traces
-
-## 4. Verify
-
-- All prompts created with the `production` label; code fetches with `label="production"`
-- Variables and subprompts compile without errors
-- Application behavior is unchanged
-- Generations show the linked prompt in the UI (if tracing enabled)

package/template/.agents/skills/integrations/langfuse/references/sdk-upgrade.md

@@ -1,34 +0,0 @@
----
-name: langfuse-sdk-upgrade
-description: Upgrade Langfuse SDKs and application instrumentation while preserving trace attributes across observations. Use for Python or JS/TS SDK migrations, including the application side of a v4 platform migration.
-metadata:
-  required_access:
-    - CODEBASE
-    - LANGFUSE_PROJECT_SCRIPT
----
-
-# Langfuse SDK upgrade
-
-## Sources of truth
-
-Determine every installed Langfuse SDK major version, then fetch each applicable leaf guide in full before editing:
-
-- [SDK upgrade paths](https://langfuse.com/docs/observability/sdk/upgrade-path)
-- [Python v3 to v4](https://langfuse.com/docs/observability/sdk/upgrade-path/python-v3-to-v4)
-- [JS/TS v4 to v5](https://langfuse.com/docs/observability/sdk/upgrade-path/js-v4-to-v5)
-- [Instrumentation and attribute propagation](https://langfuse.com/docs/observability/sdk/instrumentation#add-attributes)
-- [Sessions and session-level metrics](https://langfuse.com/docs/observability/features/sessions)
-- [Direct OpenTelemetry ingestion](https://langfuse.com/integrations/native/opentelemetry)
-
-Follow every intermediate guide when the installed version is more than one major behind. Use the current docs for implementation details; do not copy examples from this file.
-
-## Workflow
-
-1. Inventory every SDK, integration package, direct OpenTelemetry exporter, initialization site, instrumentation wrapper, lockfile, worker, script, and test that can emit Langfuse data.
-2. Find every source of correlating attributes, including `session_id`/`sessionId`, `user_id`/`userId`, tags, metadata, version, environment, and trace name. Search for the values and surrounding application concepts, not only removed SDK method names.
-3. Apply every relevant item from the exact version-specific guides. Preserve each correlating attribute by establishing its documented propagation scope before any observation-producing call that must inherit it.
-
-
-## Completion report
-
-Report the versions before and after, changed instrumentation paths, attribute sources and propagation scopes, validation performed, inspected trace or session, and any remaining blocked verification.

package/template/.agents/skills/integrations/langfuse/references/setting-up-evals.md

@@ -1,65 +0,0 @@
----
-name: langfuse-setting-up-evals
-description: Set up evaluation by finding gaps across signal capture, monitoring, and evaluator metrics, then addressing the right one first.
-metadata:
-  required_access:
-    - LANGFUSE_PROJECT_INTERFACE
----
-
-# Setting up evals
-
-- **Act as executor and teacher.** Do the work and, above all, explain why the approach fits the user's goal and data.
-- **Teach decision-making, not configuration.** Explain the reasoning, tradeoffs, and limitations—not sampling rates, targets, score types, or other setup details. The response must leave the user able to repeat or change the approach and make future decisions without you.
-
-Always read [Choosing what to evaluate](https://langfuse.com/academy/evaluate/choosing-what-to-evaluate) and [Evaluation](https://langfuse.com/academy/evaluate). Use their guidance to interpret their project, but do not return it as a generic setup plan.
-
-If the user has a precise metric, skip to **Build the evaluator**. Otherwise, do not assume that choosing evaluator metrics is the next step.
-
-## Find the measurement gap
-
-Before advising:
-
-- Inspect enough representative traces to present concrete findings of your own before asking about metrics.
-- Inventory existing datasets, evaluators, traces, etc. as context; do not assume they should be reused or represent a current priority. Treat their importance as unconfirmed until the user agrees.
-- Identify what is already surfaced through trace errors, logged data, existing scores, or other monitoring; what is available but not monitored; what important user or product signals are not captured; and what genuinely requires an evaluator.
-- Do not ask the user for information you can inspect.
-- If project access is unavailable, say so instead of substituting a generic plan.
-
-Do not add an evaluator that duplicates an existing error or logged signal. For example, do not evaluate valid JSON when invalid JSON already produces a trace error.
-
-Treat trace errors and logged failures as existing coverage. Unless the user wants operational monitoring, do not let them displace the search for subtler qualitative signals that are not yet captured. Do not propose alerts, paging, or release gates before asking how the measurement should be used.
-
-Do not propose evaluating a known failure with an owner and a planned one-time fix unless the user wants to track whether it recurs.
-
-If formal error analysis is the right next step, explain why and ask whether the user wants to do it. If they agree, run it following `references/error-analysis.md`, and do not start metric selection until the analysis is complete.
-
-## Define the metric set
-
-Existing project materials inform the questions you'll ask; it does not determine what the user values.
-
-When forming tentative recommendations:
-
-- Start with direct evidence of user or product outcomes, recurring failures visible in traces or user reactions, and important signals the application is not yet capturing.
-- Prefer quick wins: signals that are application-specific, actionable, reasonably reliable to start measuring. A good example signal is user (dis)satisfaction.
-- Recommend metrics only for problems evidenced in current data. A behavior being common or theoretically risky is not a reason to run an evaluator.
-- For live evaluation, recommend only metrics that can be scored from the available live data without ground truth; never suggest a metric that requires knowing the correct answer, expected outcome, or ideal resolution.
-- Do not propose generic starting metrics such as `helpfulness`, `quality`, `relevance`, `hallucination`, `groundedness`, `task completion`, `task success`, or `reliability`.
-- Before presenting any metric, verify that it can actually be measured using the available data and Langfuse's supported evaluator inputs. If not, identify what must be captured instead.
-- Explain why each recommendation deserves attention before the alternatives, then ask the user whether that priority matches their goals.
-
-When specific failures or hard requirements are known and metric selection is the next task, ask focused, dependent questions about which ones matter and what decisions their measurement should support. Give a recommendation grounded in the project, leave the decision to the user, and wait for their answer.
-
-Do not treat any metric as selected, choose evaluator types, ask implementation questions, or implement anything until the user has answered and explicitly confirmed the metric set.
-
-## Build the evaluator
-
-For evaluator functionality, use the unstable API endpoints.
-
-Follow [Writing good evaluators](https://langfuse.com/academy/evaluate/writing-evaluators) to choose the evaluator type; do not always default to an LLM-as-a-judge.
-
-- Before creating the evaluator, verify its target filter by fetching the observations it actually matches. Confirm the set is exactly what you intend—no duplicates that would be scored (and billed) twice.
-- Prioritize a clean score name: it is the metric that lands on every observation, so name it after what is measured (`refusal`), not after the evaluator (`refusal judge`). Keep any evaluator-mechanism wording out of the score name.
-- Explain why the chosen evaluation method measures the intended behavior, what evidence it relies on, what it cannot tell the user, and when another method would be better.
-- When methods have significant trade-offs and none is clearly superior, explain the options and let the user decide before implementation.
-- Only if an LLM-as-a-judge is the best fit, calibrate it on real examples before treating it as ready(`references/judge-calibration.md`). You can do this by running an experiment on a dataset where the prompt being tested is the LLM-as-a-judge prompt.
-- Share a link to the evaluator.

package/template/.agents/skills/integrations/langfuse/references/skill-feedback.md

@@ -1,35 +0,0 @@
----
-name: langfuse-skill-feedback
-description: Submit feedback about the Langfuse skill to its maintainers. Use when the skill's instructions are wrong, outdated, missing something, or could be improved.
-metadata:
-  required_access:
-    - LANGFUSE_PROJECT_INTERFACE
----
-
-# Skill Feedback
-
-Use this only for feedback about the skill itself, not Langfuse the product or a user's application data.
-
-1. Draft concise feedback using the following structure:
-  - **Describe your idea or feedback** (required)
-    A clear description of what went wrong or what could be improved. Include:
-    - What the user was trying to do (as `goal`)
-    - What the skill did vs what was expected
-    - Any specific instructions that were incorrect or missing
-
-  - **What would the ideal outcome look like?** (optional)
-    What the correct behavior or guidance should be.
-
-  If feedback targets existing skill, reference in `target`.
-2. If the user wants a reply, ask them to include an email address in `feedback`; use only an address they explicitly provide.
-3. Show every submitted field exactly as it will be sent and ask for explicit permission. Do not submit without approval.
-
-## Submission options
-
-Use the first available option, or the option the user prefers:
-
-1. **Authenticated Langfuse MCP, CLI, or Public API** — prefer the `submitFeedback` tool on the Langfuse MCP server. If it is unavailable, discover the current feedback operation with the Langfuse CLI schema/help and submit through the authenticated Public API. Do not ask users to paste credentials into chat.
-2. **Langfuse Docs MCP** — use its unauthenticated `submitFeedback` tool when no authenticated Langfuse interface is available.
-3. **GitHub issue or discussion** — if no MCP/CLI/API is available, or the user wants a public, trackable thread, provide a prefilled discussion link for the user to submit: `https://github.com/langfuse/skills/discussions/new?category=ideas-improvements&title=<url-encoded title>&body=<url-encoded body>`. Use `https://github.com/langfuse/skills/issues/new` if they prefer an issue.
-
-Report the Langfuse correlation ID or GitHub URL. If submission fails, give a safe error without exposing credentials or request bodies.

package/template/.agents/skills/integrations/langfuse/references/trace-evaluator-upgrade.md

@@ -1,76 +0,0 @@
----
-name: langfuse-evaluator-upgrade
-description: Upgrade legacy trace-level or dataset-item LLM-as-a-Judge evaluators to observation-level or experiment evaluators. Use when a project needs the coding handoff for evaluator migration during the v4 transition.
-metadata:
-  required_access:
-    - CODEBASE
-    - LANGFUSE_PROJECT_INTERFACE
-    - LANGFUSE_PROJECT_SCRIPT
----
-
-# Evaluator upgrade
-
-Use the [evaluator migration guide](https://langfuse.com/faq/all/llm-as-a-judge-migration),
-[v4 overview](https://langfuse.com/docs/v4), and current [Evaluation Rules](https://api.reference.langfuse.com/#tag/unstableevaluationrules)
-and [Evaluators](https://api.reference.langfuse.com/#tag/unstableevaluators) API schemas as
-the sources of truth. Read the current unstable schema before using it: only the unstable
-Evaluation Rules API returns legacy `trace` and `dataset` targets; new rules use observation
-or experiment targets.
-
-## Inventory and scope
-
-- Page through all rules and fetch each referenced evaluator. Record target, status, filters, mappings/JSONPaths, sampling, delay, time scope, evaluator, and score name.
-- If a bulk page fails, retry with `limit=1` to isolate unreadable entries; report any page that still fails as a blocker and do not claim a complete inventory.
-- Separate active, inactive, and blocked rules. Migrate only active rules that depend on new/live data. Historical-only rules (`EXISTING` without `NEW`) may be ignored; deactivation or deletion is optional and requires approval.
-- Migrate only legacy trace and dataset targets here. Do not touch existing observation, experiment, or event rules. If the unstable API cannot mutate a legacy rule, return the exact UI action.
-- Show the complete inventory and a consolidated retain/delete decision before changing project configuration.
-
-## Trace to observation
-
-In v4, a trace groups observations and has no separate trace input/output target. For each
-retained trace rule, select one stable observation and verify its own fields:
-
-1. Prefer an existing root or parent observation that already contains the required result.
-2. If the rule used overall trace input/output, confirm that the previous SDK upgrade has already put those values on the root observation.
-3. If it mixed trace context with one observation, propagate only the required context or filter attributes to that observation.
-4. If required values span multiple observations, add a dedicated evaluation observation only when no existing observation can own them. Do not serialize the whole trace into metadata.
-
-Use a stable observation name/type and root status where relevant. The selector should ideally match at
-most one observation per trace when the legacy rule produced one score per trace. If it matches multiple observation, convey to the user that the rule will produce one score per observation. Keep names
-and types stable across executions.
-
-Preserve the evaluator-template: prompt, model, output, score name, sampling, delay. Time scope, filters, mappings, and JSONPaths need to be compatible with the new format. A legacy trace
-name filter becomes `traceName`; it does not replace the observation selector. Every variable
-and filter must exist on the selected observation, with trace attributes propagated according
-to the applicable SDK or OpenTelemetry guide.
-
-Inspect the resolved SDK/OTel path and a representative new trace; do not infer readiness
-from a dependency declaration alone. Complete application instrumentation first when needed.
-Keep deprecated trace-I/O calls only while an active legacy rule requires them, and remove
-them after the successor is verified and the legacy rule is disabled.
-
-## Dataset item to experiment
-
-Read the legacy `dataset` rule and referenced evaluator, then create an `experiment` rule with
-the same evaluator, score name, sampling, and mappings. Use the experiment target's current
-dataset filter and dataset ID from the unstable schema.
-As of the current API, this is `datasetId` with IDs from the v2 dataset endpoint. Translate
-dataset-item expected output and metadata to the corresponding experiment-item fields, and
-preserve input mappings and JSONPaths against the experiment-item shape.
-
-For dataset runs, the root observation carries the run's input and output, so mapping legacy
-trace input/output to observation input/output is safe and needs no evaluator-specific
-instrumentation change.
-
-Keep the legacy rule active while the project still uses low-level dataset-run APIs. Disable
-it after the project emits experiment context; leaving both active can create duplicate
-scores. Review any mapping that relies on a named observation or assumes a one-to-one input.
-
-## Cut over and report
-
-- Create the successor disabled when supported, validate it on new data, then enable it with approval. A brief overlap can create duplicate scores.
-- Disable the legacy rule only after the successor is verified. Do not backfill historical data or delete historical scores.
-- Verify the selected observation exists once when required, contains every mapped value and filter, and excludes sibling observations. Do not claim live verification without project access or a runnable path.
-
-Return a concise per-rule table with the legacy rule, target, selector, mappings/filters,
-required code changes, validation result, and remaining project actions or blockers.