quiver-cli 1.3.0 → 2.0.0

package/template/.agents/skills/integrations/langfuse/references/judge-calibration.md

@@ -1,291 +0,0 @@
----
-name: langfuse-judge-calibration
-description: Calibrate and validate LLM-as-a-Judge evaluators against dataset ground truth. Runs the judge prompt as a Langfuse dataset experiment, compares judge outputs with dataset item expected outputs, and reports simple accuracy or advanced confusion-matrix metrics. Use this guide whenever a user asks if their LLM judge is actually useful, aligned with human judgment, or safe to trust for monitoring decisions.
-metadata:
-  required_access:
-    - CODEBASE
-    - LANGFUSE_PROJECT_SCRIPT
----
-
-# Judge Calibration (LLM-as-a-Judge)
-
-## Goal
-
-Validate judge outputs against human labels using the smallest reliable workflow
-for the user's goal.
-
-Default to a **Langfuse dataset experiment** when the user has a Langfuse
-dataset or wants results in the Langfuse Experiments UI.
-
-Default to **simple calibration** unless the user asks for deeper metrics,
-split-based validation, thresholding, or production automation.
-
-## 1) Choose the calibration mode
-
-### Simple calibration
-
-Use this when the user wants a quick answer like "does this judge basically
-match human labels?" or explicitly asks for accuracy only.
-
-- No train/dev/test split is required.
-- Compute `exact_match` for each valid row.
-- Report valid sample size, invalid-label count, accuracy, and a short
-  recommendation.
-- Do not include Precision/Recall/F1, TPR/TNR, denominator notes, or top failure
-  direction unless the user asks for advanced metrics.
-
-### Advanced calibration
-
-Use this when the user asks for confusion matrix metrics, thresholds,
-production monitoring, high-stakes automation, or train/test-style validation.
-
-- If split labels exist, keep train/dev/test separate.
-- If no split exists, compute metrics on the provided rows and state that this is
-  not a held-out final quality claim.
-- Compute TP/FP/FN/TN and derived metrics.
-- Use `references/error-analysis.md` for qualitative diagnosis of disagreements.
-
-## 2) Primary workflow
-
-1. Confirm the dataset name, ground-truth label location in `expectedOutput`,
-   judge prompt name/version, judge model, and label vocabulary.
-2. Choose simple or advanced mode. If ambiguous, use simple mode.
-3. Run the judge prompt against each dataset item input as a Langfuse experiment.
-4. Compare the judge output to `item.expected_output` in evaluator functions.
-5. Return the matching report format from section 7.
-
-## 3) Langfuse experiment workflow
-
-Use the SDK experiment runner as the default implementation. A Langfuse-hosted
-dataset automatically creates a dataset run that can be inspected and compared
-in the Langfuse UI.
-
-Before implementing, you **must** retrieve the current experiment SDK
-documentation from the Langfuse docs — do not rely on memory, the SDK changes
-frequently. Fetch these pages (see SKILL.md section 2 for retrieval methods):
-
-- [Experiments via SDK](https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk) — primary reference for `dataset.run_experiment`, task/evaluator signatures, and `Evaluation` return shape
-- [Datasets](https://langfuse.com/docs/evaluation/experiments/datasets) — dataset item structure (`input`, `expectedOutput`) and how to load a hosted dataset
-- [Experiments data model](https://langfuse.com/docs/evaluation/experiments/data-model) — how runs, items, and scores relate in the UI
-
-High-level shape of a simple-mode calibration experiment:
-
-```
-load dataset and judge prompt from Langfuse
-define POSITIVE / NEGATIVE label set
-
-task(item):
-    compile judge prompt with item.input
-    call judge model
-    return normalized label
-    # never read item.expected_output here — that would leak the answer
-
-item_evaluator(output, expected_output):
-    if either label is outside the allowed set:
-        return invalid (excluded from accuracy denominator)
-    return exact_match score (0 or 1)
-
-run_evaluator(item_results):
-    accuracy = matches / valid_rows
-    return aggregate score + invalid_label count
-
-dataset.run_experiment(task, [item_evaluator], [run_evaluator],
-                      metadata={calibration_mode, judge_prompt, labels})
-```
-
-For advanced calibration, the item evaluator emits one score per confusion-matrix
-cell (`judge-is-tp`, `judge-is-fp`, `judge-is-fn`, `judge-is-tn`), and the run
-evaluator aggregates them into Precision / Recall / F1 / TPR / TNR. See section
-6 for the metric definitions and zero-denominator guardrails.
-
-Rules:
-- Use a Langfuse-hosted dataset when the user wants a real Langfuse experiment.
-  Local SDK datasets create traces and scores, but not Langfuse dataset runs.
-- The dataset item `input` must contain everything needed to run the judge
-  prompt. The dataset item `expectedOutput` must contain the ground-truth label.
-- Never pass `expectedOutput` into the judge prompt or task. That would leak the
-  answer and invalidate calibration.
-- Return `Evaluation(...)` objects from item and run evaluators for stable SDK
-  formatting and score ingestion.
-- Store prompt name/version, judge model, label vocabulary, dataset version, and
-  calibration mode in run metadata.
-- Use a unique run name so the experiment appears as a separate dataset run.
-
-## 4) Label validation
-
-Never silently treat unknown labels as negative.
-
-- For binary judges, define the positive and negative labels before computing
-  confusion-matrix metrics.
-- Normalize only deterministic differences such as surrounding whitespace or
-  casing, and mention that normalization was applied.
-- Rows where `expected` or `actual` is outside the allowed label set are invalid.
-  Exclude them from metric denominators and report the invalid count.
-- For multi-class judges, use simple `exact_match` / accuracy unless the user
-  defines one positive class for binary metrics.
-
-Example binary labels:
-- Positive: `ESCALATE`
-- Negative: `RESOLVE`
-
-## 5) Simple metrics
-
-For each valid row:
-
-- `exact_match = 1 if actual == expected else 0`
-
-Aggregate:
-
-- `accuracy = sum(exact_match) / valid_rows`
-
-If `valid_rows == 0`, report that accuracy is undefined and ask for valid
-expected/actual labels.
-
-## 6) Advanced metrics
-
-### Dataset and split discipline
-
-Use split discipline only when the user asks for advanced validation or final
-quality claims.
-
-- **Train**: optional few-shot examples for the judge prompt
-- **Dev**: iterative prompt/model refinement
-- **Test**: single final calibration pass
-
-Do not tune on the same rows used to claim final quality. Use balanced classes
-in dev/test when possible so both error directions are measurable.
-
-### Per-row classification mapping
-
-For each row with `expected` and `actual` labels:
-
-- **TP**: expected = positive and actual = positive
-- **FP**: expected = negative and actual = positive
-- **FN**: expected = positive and actual = negative
-- **TN**: expected = negative and actual = negative
-
-Also compute:
-- `exact_match = 1 if actual == expected else 0`
-
-### Aggregate metrics
-
-From aggregate counts:
-- `accuracy = (TP + TN) / valid_rows`
-- `precision = TP / (TP + FP)`
-- `recall = TP / (TP + FN)`
-- `f1 = 2 * precision * recall / (precision + recall)`
-- `TPR = TP / (TP + FN)`
-- `TNR = TN / (TN + FP)`
-
-Guardrails:
-- if `TP + FP == 0`, precision is undefined (report null + note)
-- if `TP + FN == 0`, recall and TPR are undefined (report null + note)
-- if `TN + FP == 0`, TNR is undefined (report null + note)
-- if `precision + recall == 0`, set `f1 = 0`
-
-### Advanced quality gates
-
-Before trusting the judge on production traffic:
-
-1. **Split integrity**: no leakage from held-out rows into prompt examples.
-2. **Confusion matrix sanity**: `TP + FP + FN + TN == valid_rows`.
-3. **Metric recomputation check**: recompute aggregate stats from row-level
-   flags and compare.
-4. **TPR/TNR review**: inspect both directions for class-direction bias.
-5. **Threshold**: target `TPR > 0.90` and `TNR > 0.90` before high-stakes
-   automation.
-
-## 7) Report format
-
-### Simple report
-
-Return only:
-- dataset name and dataset run URL when available
-- valid rows / total rows
-- invalid-label count
-- accuracy
-- one-sentence recommendation
-
-### Advanced report
-
-Add:
-- confusion matrix: TP, FP, FN, TN
-- accuracy, precision, recall, F1, TPR, TNR
-- denominator notes for undefined metrics
-- top failure direction: false positives or false negatives
-- recommendation: ship, iterate, collect more labels, or do not automate
-
-## 8) Langfuse implementation notes
-
-Prefer SDK experiment evaluators for score creation. They attach item-level
-scores to the experiment traces and run-level scores to the dataset run.
-
-Use manual REST score creation only as a fallback when not using the SDK
-experiment runner, or for local smoke tests. See
-[Scores via SDK](https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk)
-and the [Scores API reference](https://langfuse.com/docs/api) (`POST /api/public/scores`)
-for the current payload shape. Do not use the current `langfuse-cli` score-create
-wrapper unless `--help` shows a usable `value` argument; `langfuse-cli@0.0.10`
-exposes `legacy-score-v1s create` but cannot pass the required score `value`.
-
-Score names to emit:
-
-Simple mode:
-- `judge-exact-match`
-- `judge-accuracy`
-
-Advanced mode:
-- `judge-exact-match`
-- `judge-is-tp`
-- `judge-is-fp`
-- `judge-is-fn`
-- `judge-is-tn`
-
-Recommended metadata:
-- `expected_label`, `actual_label`
-- calibration mode: `simple` or `advanced`
-- positive/negative labels when binary metrics are used
-- evaluator prompt name+version
-- dataset/split version when used
-- run identifier
-
-## 9) Classification logic (pseudo-code)
-
-The per-row classification each evaluator must perform:
-
-```
-normalize(label) = strip whitespace, uppercase
-ALLOWED = {POSITIVE, NEGATIVE}
-
-classify(expected, actual):
-    expected, actual = normalize(expected), normalize(actual)
-
-    if expected ∉ ALLOWED or actual ∉ ALLOWED:
-        mark row invalid → exclude from denominators
-
-    exact_match = (expected == actual)
-    is_tp = (expected == POSITIVE and actual == POSITIVE)
-    is_fp = (expected == NEGATIVE and actual == POSITIVE)
-    is_fn = (expected == POSITIVE and actual == NEGATIVE)
-    is_tn = (expected == NEGATIVE and actual == NEGATIVE)
-```
-
-## 10) Common failure modes
-
-- label vocabulary not constrained (judge outputs free text instead of strict
-  labels)
-- positive/negative label inversion between annotators and evaluator code
-- leaking `expectedOutput` into the judge task instead of only the evaluator
-- using local SDK data when the user expects a Langfuse dataset run in the UI
-- reporting only accuracy when classes are imbalanced and error direction matters
-- calculating F1 without explicit zero-denominator handling
-- using advanced validation claims without a held-out split
-
-## 11) What to do after calibration
-
-- If simple accuracy is enough: report it and stop.
-- If metrics are weak and advanced validation is needed: iterate prompt and
-  few-shots on dev data only.
-- If metrics pass: freeze the baseline and monitor drift over time.
-- For qualitative diagnosis of disagreements, switch to
-  `references/error-analysis.md`.

package/template/.agents/skills/integrations/langfuse/references/prompt-engineering.md

@@ -1,35 +0,0 @@
----
-name: langfuse-prompt-engineering
-description: Write or change prompts in Langfuse or code. Use whenever the user asks to create, edit, rewrite, debug, tune, or otherwise modify a prompt, including a small wording or instruction change. Distinct from prompt-migration and judge-calibration.
-metadata:
-  required_access: []
----
-
-# Prompt Engineering
-
-## Universal principles
-
-1. **Be specific.** State the required output format, constraints, and, when order matters, numbered steps. Write for someone with minimal context.
-2. **Use labeled sections** for the role, instructions, examples, and context. Use XML tags when they help distinguish rules, data, and variables.
-3. **Give the model a role.** A focused system-prompt sentence can steer tone and behavior.
-4. **Explain why important rules exist** so the model can apply them to adjacent cases.
-
-## Adjusting an existing prompt
-
-1. **Pin down the failure first.** Use concrete misbehaving outputs and name the exact failure mode: wrong format, ignored instruction, wrong tone, hallucination, or another observable error. The task and data show what could fail, not what did fail; do not write a fix before observing the failure.
-2. **Trace the failure to the prompt.** Look for a missing, ambiguous, or conflicting instruction.
-3. **Fix the failure class, not the example.** Generalize the observed mistake into an error class and write the instruction at that level, in your own words. Cover adjacent cases, but do not add definitions, rules, or examples for unobserved problems or encode the failing example itself. If a generalized instruction does not work, report the failure instead of narrowing the rule to that case.
-4. **Preserve existing behavior.** Before changing text, identify what behavior it protects. Avoid contradicting other rules or breaking cases that already work.
-5. **Make the smallest testable edit.** Change one cause per attempt so you can attribute the result. Avoid extra length, emphasis, or all-caps. In Langfuse, save each attempt as a new version or label so you can compare and roll back.
-
-## Model-specific tuning
-
-Identify the target model if you can — tuning differs by model, so when you know which model the prompt runs on, follow that model's own prompting guidance.
-
-| Technique | Standard (GPT) | Reasoning (Claude latest) |
-|-----------|----------------|---------------------------|
-| Instruction density | High, prescriptive | Goal-level; don't micro-manage steps |
-| Reasoning | Add explicit chain-of-thought | Native thinking; "think thoroughly" + tune `effort` |
-| Emphasis language | Strong directives fine | Dial back `MUST`/`CRITICAL` |
-| Output format | Prefilling/scaffolds | Prefill removed (4.6+); structured outputs + "no preamble" |
-| Roles | `developer` > `user` > `assistant` | `system` role + `user`/`assistant` |

package/template/.agents/skills/integrations/langfuse/references/prompt-migration.md

@@ -1,79 +0,0 @@
----
-name: langfuse-prompt-migration
-description: Migrate hardcoded prompts to Langfuse for version control and deployment-free iteration. Use when user wants to externalize prompts, move prompts to Langfuse, or set up prompt management.
-metadata:
-  required_access:
-    - CODEBASE
-    - LANGFUSE_PROJECT_SCRIPT
----
-
-# Langfuse Prompt Migration
-
-Migrate hardcoded prompts into Langfuse-managed prompts. The API mechanics (`create_prompt`, `get_prompt`, `.compile()`, linking to traces) are in the docs — fetch them at execution time.
-
-## Prerequisites
-
-Verify credentials exist — check presence only, never print the secret key (its value would land in the agent's context and transcripts):
-
-```bash
-[ -n "$LANGFUSE_PUBLIC_KEY" ] && echo "public key: set" || echo "public key: missing"
-[ -n "$LANGFUSE_SECRET_KEY" ] && echo "secret key: set" || echo "secret key: missing"
-[ -n "${LANGFUSE_BASE_URL:-${LANGFUSE_HOST:-}}" ] && echo "base url: set" || echo "base url: missing"
-```
-
-Use `LANGFUSE_BASE_URL` for current SDKs. If only `LANGFUSE_HOST` is set, export `LANGFUSE_BASE_URL="$LANGFUSE_HOST"`. If a CLI expects `LANGFUSE_HOST` and only `LANGFUSE_BASE_URL` is set, export `LANGFUSE_HOST="$LANGFUSE_BASE_URL"`.
-
-If credentials or the base URL are missing, ask the user to set them in their shell or a `.env` file. Do not ask them to paste secret keys into chat.
-
-## 1. Inventory every prompt (before writing any code)
-
-For each prompt, record:
-
-- **Name**: lowercase, hyphenated (e.g. `chat-assistant`)
-- **Source file**: where the prompt text lives
-- **Code file to refactor**: the file that USES the prompt. For asset files (`.txt`/`.yaml`/`.md`), this is the file that loads the asset, not the asset itself
-- **Type**: `chat` (message array) or `text` (plain string)
-- **Variables**: values interpolated in, converted to `{{var}}`
-- **Content**: the actual text to upload
-
-Before choosing `text` or `chat`, fetch and follow [Chat vs Text Prompts](https://langfuse.com/docs/prompt-management/data-model#text-vs-chat-prompts). Do not default unrelated prompt flows to one prompt type.
-
-Prompts typically live in OpenAI message arrays, Anthropic system arguments, LangChain prompt templates, Vercel AI system/prompt fields, and raw multi-line strings near LLM calls.
-
-## 2. Convert templating and decide structure
-
-**Variable syntax:** Langfuse substitutes only double-brace `{{var}}`. Convert every single-brace form during upload — `{var}`, `${var}`, f-string `{var}`, `.format(var=...)`, and string concatenation all become `{{var}}`. Uploading `{var}` will silently fail to substitute.
-
-**Complex templates:** Langfuse has no conditionals, loops, or filters. If the code uses them (e.g. Jinja `{% if %}`/`{% for %}`), either pre-compute the value in code and pass a plain `{{variable}}` (recommended), or store the raw template and compile client-side — which loses Playground preview and UI experiments. See https://langfuse.com/docs/prompt-management/features/variables and the [external templating FAQ](https://langfuse.com/faq/all/using-external-templating-libraries).
-
-**What to make a variable vs. keep hardcoded:**
-
-| Make variable | Keep hardcoded |
-|---------------|----------------|
-| User-specific (`{{user_name}}`) | Output format instructions |
-| Dynamic content (`{{context}}`) | Safety guardrails |
-| Per-request (`{{query}}`) | Persona / personality |
-| Environment-specific (`{{company_name}}`) | Static examples |
-
-**Naming:** lowercase-hyphenated, feature-based (`document-summarizer`), hierarchical for related prompts (`support/triage`), prefix subprompts with `_` (`_base-personality`).
-
-Before extracting subprompts, fetch and follow [Prompt Composability](https://langfuse.com/docs/prompt-management/features/composability). Use composition for the reuse and shared-maintenance cases described there, not merely to decompose one coherent prompt flow.
-
-## 3. Present the plan, then create and refactor
-
-When the user explicitly requests a migration and credentials work, treat that request as authorization to create prompts and refactor the call sites. Present the inventory and plan as a progress update, then continue.
-
-Ask only when a materially different design choice would change behavior, required credentials are unavailable, or destructive cleanup needs approval. Do not stop merely to confirm names, prompt types, or optional tracing.
-
-Then:
-
-- Create the prompts (label migrated prompts `production` — they're already live) and refactor call sites to fetch each prompt from Langfuse and compile its variables in. The SDK calls differ across Python and JS/TS — fetch the current docs: https://langfuse.com/docs/prompt-management/get-started
-- Fetch by the `production` label
-- If the codebase already has Langfuse tracing (decorators, an instrumented client, or manual spans), link prompts so you can see which version produced each response. See https://langfuse.com/docs/prompt-management/features/link-to-traces
-
-## 4. Verify
-
-- All prompts created with the `production` label; code fetches with `label="production"`
-- Variables and subprompts compile without errors
-- Application behavior is unchanged
-- Generations show the linked prompt in the UI (if tracing enabled)

package/template/.agents/skills/integrations/langfuse/references/skill-feedback.md

@@ -1,55 +0,0 @@
----
-name: langfuse-skill-feedback
-description: Submit feedback about the Langfuse skill to its maintainers via GitHub Discussions. Use when the user indicates the skill gave incorrect guidance, is missing information, or could be improved.
-metadata:
-  required_access:
-    - GITHUB
----
-
-# Skill Feedback
-
-Follow these steps exactly:
-
-1. **Ask permission**: Ask the user if they'd like you to submit feedback to the skill maintainers. Make it clear this is about the skill (the agent instructions), not about Langfuse the product. If they decline, move on.
-2. **Draft feedback**: Write the feedback using the form structure below. Present the draft to the user and ask if they'd like to change anything before submitting.
-3. **Submit**: Once approved, submit via `gh` CLI as described below. Share the resulting discussion URL with the user.
-
-## Feedback Form Structure
-
-Draft the feedback using these two fields:
-
-**Describe your idea or feedback** (required)
-A clear description of what went wrong or what could be improved. Include:
-- What the user was trying to do
-- What the skill did vs what was expected
-- Any specific instructions that were incorrect or missing
-
-**What would the ideal outcome look like?** (optional)
-What the correct behavior or guidance should be.
-
-Format the body as markdown with the two field labels as headings.
-
-## Submitting
-
-Create a GitHub Discussion on the `langfuse/skills` repository using the GraphQL API:
-
-```bash
-gh api graphql -f query='
-mutation($repoId: ID!, $categoryId: ID!, $title: String!, $body: String!) {
-  createDiscussion(input: {repositoryId: $repoId, categoryId: $categoryId, title: $title, body: $body}) {
-    discussion { url }
-  }
-}' \
-  -f repoId="$(gh api graphql -f query='{ repository(owner: "langfuse", name: "skills") { id } }' --jq '.data.repository.id')" \
-  -f categoryId="$(gh api graphql -f query='{ repository(owner: "langfuse", name: "skills") { discussionCategories(first: 10) { nodes { id name } } } }' --jq '.data.repository.discussionCategories.nodes[] | select(.name == "Ideas & Improvements") | .id')" \
-  -f title="<concise title>" \
-  -f body="<formatted feedback>"
-```
-
-If the `gh` CLI is not authenticated or the request fails, give the user this link to create the discussion manually:
-
-```
-https://github.com/langfuse/skills/discussions/new?category=ideas-improvements
-```
-
-After submission, share the discussion URL with the user.

package/template/.agents/skills/integrations/langfuse/references/trace-evaluator-upgrade.md

@@ -1,76 +0,0 @@
----
-name: langfuse-evaluator-upgrade
-description: Upgrade legacy trace-level or dataset-item LLM-as-a-Judge evaluators to observation-level or experiment evaluators. Use when a project needs the coding handoff for evaluator migration during the v4 transition.
-metadata:
-  required_access:
-    - CODEBASE
-    - LANGFUSE_PROJECT_INTERFACE
-    - LANGFUSE_PROJECT_SCRIPT
----
-
-# Evaluator upgrade
-
-Use the [evaluator migration guide](https://langfuse.com/faq/all/llm-as-a-judge-migration),
-[v4 overview](https://langfuse.com/docs/v4), and current [Evaluation Rules](https://api.reference.langfuse.com/#tag/unstableevaluationrules)
-and [Evaluators](https://api.reference.langfuse.com/#tag/unstableevaluators) API schemas as
-the sources of truth. Read the current unstable schema before using it: only the unstable
-Evaluation Rules API returns legacy `trace` and `dataset` targets; new rules use observation
-or experiment targets.
-
-## Inventory and scope
-
-- Page through all rules and fetch each referenced evaluator. Record target, status, filters, mappings/JSONPaths, sampling, delay, time scope, evaluator, and score name.
-- If a bulk page fails, retry with `limit=1` to isolate unreadable entries; report any page that still fails as a blocker and do not claim a complete inventory.
-- Separate active, inactive, and blocked rules. Migrate only active rules that depend on new/live data. Historical-only rules (`EXISTING` without `NEW`) may be ignored; deactivation or deletion is optional and requires approval.
-- Migrate only legacy trace and dataset targets here. Do not touch existing observation, experiment, or event rules. If the unstable API cannot mutate a legacy rule, return the exact UI action.
-- Show the complete inventory and a consolidated retain/delete decision before changing project configuration.
-
-## Trace to observation
-
-In v4, a trace groups observations and has no separate trace input/output target. For each
-retained trace rule, select one stable observation and verify its own fields:
-
-1. Prefer an existing root or parent observation that already contains the required result.
-2. If the rule used overall trace input/output, confirm that the previous SDK upgrade has already put those values on the root observation.
-3. If it mixed trace context with one observation, propagate only the required context or filter attributes to that observation.
-4. If required values span multiple observations, add a dedicated evaluation observation only when no existing observation can own them. Do not serialize the whole trace into metadata.
-
-Use a stable observation name/type and root status where relevant. The selector should ideally match at
-most one observation per trace when the legacy rule produced one score per trace. If it matches multiple observation, convey to the user that the rule will produce one score per observation. Keep names
-and types stable across executions.
-
-Preserve the evaluator-template: prompt, model, output, score name, sampling, delay. Time scope, filters, mappings, and JSONPaths need to be compatible with the new format. A legacy trace
-name filter becomes `traceName`; it does not replace the observation selector. Every variable
-and filter must exist on the selected observation, with trace attributes propagated according
-to the applicable SDK or OpenTelemetry guide.
-
-Inspect the resolved SDK/OTel path and a representative new trace; do not infer readiness
-from a dependency declaration alone. Complete application instrumentation first when needed.
-Keep deprecated trace-I/O calls only while an active legacy rule requires them, and remove
-them after the successor is verified and the legacy rule is disabled.
-
-## Dataset item to experiment
-
-Read the legacy `dataset` rule and referenced evaluator, then create an `experiment` rule with
-the same evaluator, score name, sampling, and mappings. Use the experiment target's current
-dataset filter and dataset ID from the unstable schema.
-As of the current API, this is `datasetId` with IDs from the v2 dataset endpoint. Translate
-dataset-item expected output and metadata to the corresponding experiment-item fields, and
-preserve input mappings and JSONPaths against the experiment-item shape.
-
-For dataset runs, the root observation carries the run's input and output, so mapping legacy
-trace input/output to observation input/output is safe and needs no evaluator-specific
-instrumentation change.
-
-Keep the legacy rule active while the project still uses low-level dataset-run APIs. Disable
-it after the project emits experiment context; leaving both active can create duplicate
-scores. Review any mapping that relies on a named observation or assumes a one-to-one input.
-
-## Cut over and report
-
-- Create the successor disabled when supported, validate it on new data, then enable it with approval. A brief overlap can create duplicate scores.
-- Disable the legacy rule only after the successor is verified. Do not backfill historical data or delete historical scores.
-- Verify the selected observation exists once when required, contains every mapped value and filter, and excludes sibling observations. Do not claim live verification without project access or a runnable path.
-
-Return a concise per-rule table with the legacy rule, target, selector, mappings/filters,
-required code changes, validation result, and remaining project actions or blockers.

package/template/.agents/skills/integrations/langfuse/references/user-feedback.md

@@ -1,92 +0,0 @@
----
-name: langfuse-user-feedback
-description: Wires up user feedback (thumbs up/down, ratings, comments) from an application's frontend to Langfuse scores. Use when user wants to capture end-user feedback, add ratings to traces, or connect user complaints to Langfuse.
-metadata:
-  required_access:
-    - CODEBASE
-    - LANGFUSE_PROJECT_SCRIPT
----
-
-# User Feedback
-
-Tracing must already be set up — feedback is stored as scores on traces.
-
-Docs: https://langfuse.com/docs/observability/features/user-feedback
-
-## Workflow
-
-### 1. Determine What Feedback to Capture
-
-If the user has asked for something specific, go with that. Otherwise, look at the application and **present a few UX options** for how feedback could work, then ask the user which they prefer before implementing.
-
-Common UX patterns to suggest:
-
-| UX Pattern | Best for | How it works |
-|------------|----------|--------------|
-| Thumbs up/down | Chat apps, Q&A | Simple binary buttons next to each response |
-| Star rating (1–5) | Content generation, summaries | Star row or dropdown after each output |
-| "Was this helpful?" banner | Search, documentation assistants | Single yes/no prompt at the bottom of a response |
-| Regenerate / copy tracking | Any app with these actions | Implicit — log when users retry (negative signal) or copy output (positive signal) |
-| Free-text comment | Complex outputs, internal tools | Optional text field alongside a rating |
-| Report button | Any user-facing app | Flag icon to report bad/harmful responses |
-
-This table is not exhaustive — if the application suggests a different feedback pattern that fits better, propose that instead. Present 2–3 options that match the application's use case and ask the user which approach they'd like. This decision shapes everything downstream (score names, data types, frontend components), so it's important to align early.
-
-Feedback can be **explicit** (user rates via thumbs, stars, etc.) or **implicit** (derived from behavior like copying output, retrying, or escalating to support). Both are stored as scores. Explicit feedback requires the trace ID to reach the frontend; implicit feedback is logged server-side where the event already happens.
-
-### 2. Choose Score Names
-
-Name reflects the signal source, not what you hope it measures (e.g., `user-thumbs` not `response-quality` — a thumbs down doesn't tell you *what* was wrong). Avoid generic names like `feedback` or `score`.
-
-Rules:
-- Lowercase with hyphens
-- One consistent name per feedback type across the entire app
-- If capturing multiple signals, each gets its own distinct name
-
-### 3. Implement Score Creation
-
-**For implicit feedback (server-side):** Use `langfuse.create_score()` / `langfuse.score.create()` wherever the event is already handled in application code. Fetch SDK docs for current API: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk
-
-**For explicit feedback (frontend):** Use `LangfuseWeb` in the browser. It uses the public key only — no secret key exposed.
-
-```typescript
-import { LangfuseWeb } from "langfuse";
-
-const langfuse = new LangfuseWeb({
-  publicKey: process.env.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY!,
-  baseUrl: process.env.NEXT_PUBLIC_LANGFUSE_HOST,
-});
-
-langfuse.score({
-  traceId,
-  name: "user-thumbs",
-  value: 1, // 1 = positive, 0 = negative
-  dataType: "BOOLEAN",
-  comment: optionalUserComment,
-});
-```
-
-The trace ID must be available in the frontend for this to work. For Vercel AI SDK, the non-obvious pattern is using `generateMessageId`:
-
-```typescript
-import { getActiveTraceId } from "@langfuse/tracing";
-
-// Inside route handler wrapped with observe()
-return result.toUIMessageStreamResponse({
-  generateMessageId: () => getActiveTraceId() || crypto.randomUUID(),
-});
-```
-
-### 4. Verify
-
-Trigger a feedback action and check the trace's Scores tab in Langfuse. Confirm the score name, value, and data type are correct.
-
-Point users to what they can do with feedback data: filter traces by low scores, use score analytics for trends, build annotation queues for team review.
-
-## Common Mistakes
-
-| Mistake | Problem | Fix |
-|---------|---------|-----|
-| Secret key in frontend code | Security risk | Use `LangfuseWeb` with public key only |
-| Missing `dataType` on boolean scores | Value `1` inferred as `NUMERIC` | Always pass `dataType: "BOOLEAN"` explicitly |
-| Inconsistent score names across the app | Can't aggregate or filter reliably | Pick one name per feedback type, use it everywhere |