pi-langfuse 1.4.0 → 1.4.2

package/.agents/skills/langfuse/references/cli.md

@@ -1,51 +0,0 @@
-# Langfuse CLI Reference
-
-Documentation: https://langfuse.com/docs/api-and-data-platform/features/cli
-
-## Install
-
-```bash
-# Run directly (recommended)
-npx langfuse-cli api <resource> <action>
-bunx langfuse-cli api <resource> <action>
-
-# Or install globally
-npm i -g langfuse-cli
-langfuse api <resource> <action>
-```
-
-## Discovery
-
-```bash
-# List all resources and auth info
-langfuse api __schema
-
-# List actions for a resource
-langfuse api <resource> --help
-
-# Show args/options for a specific action
-langfuse api <resource> <action> --help
-
-# Preview the curl command without executing
-langfuse api <resource> <action> --curl
-```
-
-## Credentials
-
-Set environment variables:
-
-```bash
-export LANGFUSE_PUBLIC_KEY=pk-lf-...
-export LANGFUSE_SECRET_KEY=sk-lf-...
-export LANGFUSE_HOST=https://cloud.langfuse.com  
-```
-
-## Tips
-
-- Use `--json` for machine-readable JSON output
-- Use `--curl` to preview the HTTP request without executing
-- Pagination: use `--limit` and `--page` on list endpoints
-- All list commands support filtering — check `<resource> <action> --help` for available options
-- Prefer `observations-v2s` over `observations` — the v2 endpoint returns richer data
-- Prefer `metrics-v2s` over `metrics` — the v2 endpoint returns richer data
-- Prefer `score-v2s` over `scores` — the v1 `scores` resource only supports create/delete; use `score-v2s` for list and get operations

package/.agents/skills/langfuse/references/error-analysis.md

@@ -1,100 +0,0 @@
----
-name: langfuse-error-analysis
-description: Deep-dive error analysis of an LLM pipeline or AI application using Langfuse traces.
-  Use this skill whenever the user wants to understand why their AI system is producing
-  bad outputs, where their pipeline is failing, how to categorise or label failures,
-  what to prioritise fixing, or how to set up evaluators. Also trigger for "review my
-  traces", "my outputs look wrong", "help me debug my LLM app", "I want to analyse
-  errors", "build a failure taxonomy", "what's going wrong with my pipeline", or any
-  request to systematically inspect, annotate, or score Langfuse traces. If the user
-  is trying to understand or improve the quality of an AI system's outputs, use this skill.
----
-
-# Error Analysis
-
-## Primary Guide
-
-**1. Fetch the guide in this blogpost**
-
-https://langfuse.com/guides/cookbook/error-analysis-llm-applications.md
-
-If fetch is not available query for langfuse.com error analysis guide
-
-Read it in full. It defines the authoritative 5-step process (sample selection → open coding → clustering → labelling → deciding what to fix).
-
-**2. Guide the user through this step by step**
-
-You as a coding agent and the user go through this together to perform a full error analysis with their data in langfuse. Do everything you can achieve via CLI (look up traces, create annotation queues, ...) for the user. Provide them with direct links to UI wherever their action is required. Be proactive and narrate what is going on for the user. 
-
-## Rules CRITICAL
-Use Langfuse CLI wherever possible
-Use charts where possible to display data
-
----
-
-## Langfuse Implementation Notes
-
-The guide describes the process. These notes cover the Langfuse-specific API and CLI mechanics required to execute it.
-
-### Credentials
-
-```bash
-echo $LANGFUSE_PUBLIC_KEY   # pk-lf-...
-echo $LANGFUSE_SECRET_KEY   # sk-lf-...
-echo $LANGFUSE_HOST         # https://cloud.langfuse.com (EU), https://us.cloud.langfuse.com (US), https://jp.cloud.langfuse.com (JP) or self-hosted
-```
-
-If not set, check `.env` in the project root: `export $(grep -v '^#' .env | xargs)`. If `LANGFUSE_BASE_URL` is used instead of `LANGFUSE_HOST`, run `export LANGFUSE_HOST="$LANGFUSE_BASE_URL"`.
-
-```bash
-AUTH=$(echo -n "${LANGFUSE_PUBLIC_KEY}:${LANGFUSE_SECRET_KEY}" | base64)
-
-# Verify before proceeding
-STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
-  -H "Authorization: Basic $AUTH" \
-  "${LANGFUSE_HOST}/api/public/projects")
-echo "Auth check: $STATUS"
-```
-
-If status is not `200`, stop and ask the user to check their credentials and host before continuing.
-
-### Annotation target: OBSERVATION not TRACE
-
-> **CRITICAL:** In OpenTelemetry-instrumented apps, trace-level `input`/`output` can be null — content lives in a GENERATION observation. Always add `objectType: OBSERVATION` pointing to the GENERATION observation ID to annotation queues. Adding `objectType: TRACE` shows nothing in the UI.
-
-### Annotation queues
-
-> **CRITICAL:** Queues cannot be updated or deleted after creation. Create score configs first, then the queue with all config IDs. To add new configs later, create a new queue.
-
-
-**Always give the user a direct link immediately after creating a queue:**
-
-| Host | URL pattern |
-|------|-------------|
-| EU cloud | `https://cloud.langfuse.com/project/<projectId>/annotation-queues/<queueId>` |
-| US cloud | `https://us.cloud.langfuse.com/project/<projectId>/annotation-queues/<queueId>` |
-| Self-hosted | `<LANGFUSE_HOST>/project/<projectId>/annotation-queues/<queueId>` |
-
-Instruction to give: *"Please open code the first ~50 examples. For each trace, write what you observe in the `open_coding` field (describe behaviour, don't diagnose root causes), then set `pass_fail_assessment` to Pass or Fail."*
-
-
-### Prompt fixes
-
-When a category warrants a prompt fix, always offer the user two options:
-1. Create it as a versioned prompt in Langfuse (tracked, usable via the prompt API)
-2. Draft the specific text change for them to review and apply
-
-### Setup evaluators
-
-When a category warrants an evaluator setup, propose the type of evaluator and offer to set it up for user via CLI
-
-
-### Common gotchas
-
-| Mistake | Fix |
-|---------|-----|
-| `objectType: TRACE` in queue | Use `objectType: OBSERVATION` with GENERATION obs ID |
-| Creating score config without checking existing | `GET /api/public/score-configs` first; can't delete |
-| Queue created before score configs | Create configs → collect IDs → create queue |
-| `--limit` > 100 on traces list | API hard cap; paginate with `--page` |
-| No rate limiting on queue item creation | `sleep 0.4` between calls to avoid 429 |

package/.agents/skills/langfuse/references/instrumentation.md

@@ -1,140 +0,0 @@
----
-name: langfuse-observability
-description: Instrument LLM applications with Langfuse tracing. Use when setting up Langfuse, adding observability to LLM calls, or auditing existing instrumentation.
----
-
-# Langfuse Observability
-
-Instrument LLM applications with Langfuse tracing, following best practices and tailored to your use case.
-
-## When to Use
-
-- Setting up Langfuse in a new project
-- Auditing existing Langfuse instrumentation
-- Adding observability to LLM calls
-
-## Workflow
-
-### 1. Assess Current State
-
-Check the project:
-
-- Is Langfuse SDK installed?
-- What LLM frameworks are used? (OpenAI SDK, LangChain, LlamaIndex, Vercel AI SDK, etc.)
-- Is there existing instrumentation?
-
-**No integration yet:** Set up Langfuse using a framework integration if available. Integrations capture more context automatically and require less code than manual instrumentation.
-
-**Integration exists:** Audit against baseline requirements below.
-
-### 2. Verify Baseline Requirements
-
-Every trace should have these fundamentals:
-
-| Requirement               | Check                                                                                    | Why                                                    |
-| ------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------ |
-| Model name                | Is the LLM model captured?                                                               | Enables model comparison and filtering                 |
-| Token usage               | Are input/output tokens tracked?                                                         | Enables automatic cost calculation                     |
-| Good trace names          | Are names descriptive? (`chat-response`, not `trace-1`)                                  | Makes traces findable and filterable                   |
-| Span hierarchy            | Are multi-step operations nested properly?                                               | Shows which step is slow or failing                    |
-| Correct observation types | Are generations marked as generations?                                                   | Enables model-specific analytics                       |
-| Sensitive data masked     | Is PII/confidential data excluded or masked?                                             | Prevents data leakage                                  |
-| Trace input/output        | Does the trace capture meaningful input/output? Is input explicitly set to show only relevant data (e.g., user message), not all function args? | Makes traces readable in the UI and avoids leaking sensitive args |
-
-Framework integrations (OpenAI, LangChain, etc.) handle model name, tokens, and observation types automatically. Prefer integrations over manual instrumentation.
-
-Docs: https://langfuse.com/docs/tracing
-
-### 3. Explore Traces First
-
-Once baseline instrumentation is working, encourage the user to explore their traces in the Langfuse UI before adding more context:
-
-"Your traces are now appearing in Langfuse. Take a look at a few of them—see what data is being captured, what's useful, and what's missing. This will help us decide what additional context to add."
-
-This helps the user:
-
-- Understand what they're already getting
-- Form opinions about what's missing
-- Ask better questions about what they need
-
-### 4. Discover Additional Context Needs
-
-Determine what additional instrumentation would be valuable. **Infer from code when possible, only ask when unclear.**
-
-**Infer from code:**
-
-| If you see in code...                                | Infer             | Suggest                   |
-| ---------------------------------------------------- | ----------------- | ------------------------- |
-| Conversation history, chat endpoints, message arrays | Multi-turn app    | `session_id`              |
-| User authentication, `user_id` variables             | User-aware app    | `user_id` on traces       |
-| Multiple distinct endpoints/features                 | Multi-feature app | `feature` tag             |
-| Customer/tenant identifiers                          | Multi-tenant app  | `customer_id` or tier tag |
-| Feedback collection, ratings                         | Has user feedback | Capture as scores         |
-
-**Only ask when not obvious from code:**
-
-- "How do you know when a response is good vs bad?" → Determines scoring approach
-- "What would you want to filter by in a dashboard?" → Surfaces non-obvious tags
-- "Are there different user segments you'd want to compare?" → Customer tiers, plans, etc.
-
-**Additions and their value:**
-
-| Addition            | Why                                         | Docs                                                |
-| ------------------- | ------------------------------------------- | --------------------------------------------------- |
-| `session_id`        | Groups conversations together               | https://langfuse.com/docs/tracing-features/sessions |
-| `user_id`           | Enables user filtering and cost attribution | https://langfuse.com/docs/tracing-features/users    |
-| User feedback score | Enables quality filtering and trends        | https://langfuse.com/docs/scores/overview           |
-| `feature` tag       | Per-feature analytics                       | https://langfuse.com/docs/tracing-features/tags     |
-| `customer_tier` tag | Cost/quality breakdown by segment           | https://langfuse.com/docs/tracing-features/tags     |
-
-These are NOT baseline requirements—only add what's relevant based on inference or user input.
-
-### 5. Guide to UI
-
-After adding context, point users to relevant UI features:
-
-- Traces view: See individual requests
-- Sessions view: See grouped conversations (if session_id added)
-- Dashboard: Build filtered views using tags
-- Scores: Filter by quality metrics
-
-## Framework Integrations
-
-Prefer these over manual instrumentation:
-
-| Framework     | Integration            | Docs                                                 |
-| ------------- | ---------------------- | ---------------------------------------------------- |
-| OpenAI SDK    | Drop-in replacement    | https://langfuse.com/docs/integrations/openai        |
-| LangChain     | Callback handler       | https://langfuse.com/docs/integrations/langchain     |
-| LlamaIndex    | Callback handler       | https://langfuse.com/docs/integrations/llama-index   |
-| Vercel AI SDK | OpenTelemetry exporter | https://langfuse.com/docs/integrations/vercel-ai-sdk |
-| LiteLLM       | Callback or proxy      | https://langfuse.com/docs/integrations/litellm       |
-
-Full list: https://langfuse.com/docs/integrations
-
-## Always Explain Why
-
-When suggesting additions, explain the user benefit:
-
-```
-"I recommend adding session_id to your traces.
-
-Why: This groups messages from the same conversation together.
-You'll be able to see full conversation flows in the Sessions view,
-making it much easier to debug multi-turn interactions.
-
-Learn more: https://langfuse.com/docs/tracing-features/sessions"
-```
-
-## Common Mistakes
-
-| Mistake                                        | Problem                                             | Fix                                                                               |
-| ---------------------------------------------- | --------------------------------------------------- | --------------------------------------------------------------------------------- |
-| No `flush()` in scripts                        | Traces never sent                                   | Call `langfuse.flush()` before exit                                               |
-| Flat traces                                    | Can't see which step failed                         | Use nested spans for distinct steps                                               |
-| Generic trace names                            | Hard to filter                                      | Use descriptive names: `chat-response`, `doc-summary`                             |
-| Logging sensitive data                         | Data leakage risk                                   | Mask PII before tracing                                                           |
-| Not explicitly setting input with `@observe`   | All function args become trace input (including API keys, configs) | Python: use `langfuse.update_current_span(input=...)`. JS/TS: use `updateActiveObservation({ input: ... })`. Set only the relevant input (e.g., user message) |
-| Manual instrumentation when integration exists | More code, less context                             | Use framework integration                                                         |
-| Langfuse import before env vars loaded         | Langfuse initializes with missing/wrong credentials | Import Langfuse AFTER loading environment variables (e.g., after `load_dotenv()`) |
-| Wrong import order with OpenAI                 | Langfuse can't patch the OpenAI client              | Import Langfuse and call its setup BEFORE importing OpenAI client                 |

package/.agents/skills/langfuse/references/prompt-migration.md

@@ -1,234 +0,0 @@
----
-name: langfuse-prompt-migration
-description: Migrate hardcoded prompts to Langfuse for version control and deployment-free iteration. Use when user wants to externalize prompts, move prompts to Langfuse, or set up prompt management.
----
-
-# Langfuse Prompt Migration
-
-Migrate hardcoded prompts to Langfuse for version control, A/B testing, and deployment-free iteration.
-
-## Prerequisites
-
-Verify credentials are set before starting. Check existence only — never print the secret key, since the value would land in the agent's context and transcripts:
-
-```bash
-[ -n "$LANGFUSE_PUBLIC_KEY" ] && echo "LANGFUSE_PUBLIC_KEY: set" || echo "LANGFUSE_PUBLIC_KEY: missing"
-[ -n "$LANGFUSE_SECRET_KEY" ] && echo "LANGFUSE_SECRET_KEY: set" || echo "LANGFUSE_SECRET_KEY: missing"
-[ -n "$LANGFUSE_HOST" ]       && echo "LANGFUSE_HOST: $LANGFUSE_HOST" || echo "LANGFUSE_HOST: missing"
-```
-
-If not set, ask the user to configure them in their shell or a `.env` file. Do not ask them to paste keys into chat.
-
-## Migration Flow
-
-```
-1. Scan codebase for prompts
-2. Analyze templating compatibility
-3. Propose structure (names, subprompts, variables)
-4. User approves
-5. Create prompts in Langfuse
-6. Refactor code to use get_prompt()
-7. Link prompts to traces (if tracing enabled)
-8. Verify application works
-```
-
-## Step 1: Find Prompts and Build an Inventory
-
-Before writing ANY code, make a complete list of every prompt you found. For each one, note:
-
-- Name: descriptive, lowercase, hyphenated (e.g. chat-assistant, email-classifier)
-- Source file: where the prompt text lives
-- Code file to refactor: the Python/JS file that USES the prompt (for asset files like .txt/.yaml/.md, this is the file that reads/loads the asset — NOT the asset file itself)
-- Type: chat (used as a message in a chat API) or text (used as a plain string)
-- Variables: values interpolated into the prompt, converted to {{var}} syntax:
-f-string {var} → {{var}}
-.format(var=...) → {{var}}
-${var} → {{var}}
-String concatenation + var + → {{var}}
-YAML {var} → {{var}}
-- Prompt content: the actual text to upload, with variables converted to {{var}} syntax
-
-Search for these patterns:
-
-| Framework | Look for |
-|-----------|----------|
-| OpenAI | `messages=[{"role": "system", "content": "..."}]` |
-| Anthropic | `system="..."` |
-| LangChain | `ChatPromptTemplate`, `SystemMessage` |
-| Vercel AI | `system: "..."`, `prompt: "..."` |
-| Raw | Multi-line strings near LLM calls |
-
-## Step 2: Check Templating Compatibility
-
-**CRITICAL:** Langfuse only supports simple `{{variable}}` substitution. No conditionals, loops, or filters.
-
-| Template Feature | Langfuse Native | Action |
-|------------------|-----------------|--------|
-| `{{variable}}` | ✅ | Direct migration |
-| `{var}` / `${var}` | ⚠️ | Convert to `{{var}}` |
-| `{% if %}` / `{% for %}` | ❌ | Move logic to code |
-| `{{ var \| filter }}` | ❌ | Apply filter in code |
-
-**CRITICAL — Variable syntax:** Langfuse uses DOUBLE curly braces for variables: `{{var}}`. When uploading prompt content, you MUST convert every single-brace `{var}` from the original code to double-brace `{{var}}`. Never upload `{var}` — it must be `{{var}}`.
-
-### Decision Tree
-
-```
-Contains {% if %}, {% for %}, or filters?
-├─ No → Direct migration
-└─ Yes → Choose:
-    ├─ Option A (RECOMMENDED): Move logic to code, pass pre-computed values
-    └─ Option B: Store raw template, compile client-side with Jinja2
-        └─ ⚠️ Loses: Playground preview, UI experiments
-```
-
-### Simplifying Complex Templates
-
-**Conditionals** → Pre-compute in code:
-```python
-# Instead of {% if user.is_premium %}...{% endif %} in prompt
-# Use {{tier_message}} and compute value in code before compile()
-```
-
-**Loops** → Pre-format in code:
-```python
-# Instead of {% for tool in tools %}...{% endfor %} in prompt
-# Use {{tools_list}} and format the list in code before compile()
-```
-
-For external templating details, fetch: https://langfuse.com/faq/all/using-external-templating-libraries
-
-## Step 3: Propose Structure
-
-### Naming Conventions
-
-| Rule | Example | Bad |
-|------|---------|-----|
-| Lowercase, hyphenated | `chat-assistant` | `ChatAssistant_v2` |
-| Feature-based | `document-summarizer` | `prompt1` |
-| Hierarchical for related | `support/triage` | `supportTriage` |
-| Prefix subprompts with `_` | `_base-personality` | `shared-personality` |
-
-### Identify Subprompts
-
-Extract when:
-- Same text in 2+ prompts
-- Represents distinct component (personality, safety rules, format)
-- Would need to change together
-
-### Variable Extraction
-
-| Make Variable | Keep Hardcoded |
-|---------------|----------------|
-| User-specific (`{{user_name}}`) | Output format instructions |
-| Dynamic content (`{{context}}`) | Safety guardrails |
-| Per-request (`{{query}}`) | Persona/personality |
-| Environment-specific (`{{company_name}}`) | Static examples |
-
-## Step 4: Present Plan to User
-
-Format:
-```
-Found N prompts across M files:
-
-src/chat.py:
-  - System prompt (47 lines) → 'chat-assistant'
-
-src/support/triage.py:
-  - Triage prompt (34 lines) → 'support/triage'
-    ⚠️ Contains {% if %} - will simplify
-
-Subprompts to extract:
-  - '_base-personality' - used by: chat-assistant, support/triage
-
-Variables to add:
-  - {{user_name}} - hardcoded in 2 prompts
-
-Proceed?
-```
-
-## Step 5: Create Prompts in Langfuse
-
-Use `langfuse.create_prompt()` with:
-- `name`: Your chosen name
-- `prompt`: Template text (or message array for chat type)
-- `type`: `"text"` or `"chat"`
-- `labels`: `["production"]` (they're already live)
-- `config`: Optional model settings
-
-**Labeling strategy:**
-- `production` → All migrated prompts
-- `staging` → Add later for testing
-- `latest` → Auto-applied by Langfuse
-
-For full API: fetch https://langfuse.com/docs/prompts/get-started
-
-## Step 6: Refactor Code
-
-Replace hardcoded prompts with:
-
-```python
-prompt = langfuse.get_prompt("name", label="production")
-messages = prompt.compile(var1=value1, var2=value2)
-```
-
-**Key points:**
-- Always use `label="production"` (not `latest`) for stability
-- Call `.compile()` to substitute variables
-- For chat prompts, result is message array ready for API
-
-For SDK examples (Python/JS/TS): fetch https://langfuse.com/docs/prompts/get-started
-
-## Step 7: Link Prompts to Traces
-
-If codebase uses Langfuse tracing, link prompts so you can see which version produced each response.
-
-### Detect Existing Tracing
-
-Look for:
-- `@observe()` decorators
-- `langfuse.trace()` calls
-- `from langfuse.openai import openai` (instrumented client)
-
-### Link Methods
-
-| Setup | How to Link |
-|-------|-------------|
-| `@observe()` decorator | `langfuse_context.update_current_observation(prompt=prompt)` |
-| Manual tracing | `trace.generation(prompt=prompt, ...)` |
-| OpenAI integration | `openai.chat.completions.create(..., langfuse_prompt=prompt)` |
-
-### Verify in UI
-
-1. Go to **Traces** → select a trace
-2. Click on **Generation**
-3. Check **Prompt** field shows name and version
-
-For tracing details: fetch https://langfuse.com/docs/prompts/get-started#link-with-langfuse-tracing
-
-## Step 8: Verify Migration
-
-### Checklist
-
-- [ ] All prompts created with `production` label
-- [ ] Code fetches with `label="production"`
-- [ ] Variables compile without errors
-- [ ] Subprompts resolve correctly
-- [ ] Application behavior unchanged
-- [ ] Generations show linked prompt in UI (if tracing)
-
-### Common Issues
-
-| Issue | Solution |
-|-------|----------|
-| `PromptNotFoundError` | Check name spelling |
-| Variables not replaced | Use `{{var}}` not `{var}`, call `.compile()` |
-| Subprompt not resolved | Must exist with same label |
-| Old prompt cached | Restart app |
-
-## Out of Scope
-
-- Prompt engineering (writing better prompts)
-- Evaluation setup
-- A/B testing workflow
-- Non-LLM string templates