jumpstart-mode 1.1.13 → 1.1.14

package/.jumpstart/skills/langfuse/references/prompt-migration.md

@@ -0,0 +1,234 @@
+---
+name: langfuse-prompt-migration
+description: Migrate hardcoded prompts to Langfuse for version control and deployment-free iteration. Use when user wants to externalize prompts, move prompts to Langfuse, or set up prompt management.
+---
+
+# Langfuse Prompt Migration
+
+Migrate hardcoded prompts to Langfuse for version control, A/B testing, and deployment-free iteration.
+
+## Prerequisites
+
+Verify credentials before starting:
+
+```bash
+echo $LANGFUSE_PUBLIC_KEY   # pk-...
+echo $LANGFUSE_SECRET_KEY   # sk-...
+echo $LANGFUSE_HOST         # https://cloud.langfuse.com or self-hosted
+```
+
+If not set, ask user to configure them first.
+
+## Migration Flow
+
+```
+1. Scan codebase for prompts
+2. Analyze templating compatibility
+3. Propose structure (names, subprompts, variables)
+4. User approves
+5. Create prompts in Langfuse
+6. Refactor code to use get_prompt()
+7. Link prompts to traces (if tracing enabled)
+8. Verify application works
+```
+
+## Step 1: Find Prompts and Build an Inventory
+
+Before writing ANY code, make a complete list of every prompt you found. For each one, note:
+
+- Name: descriptive, lowercase, hyphenated (e.g. chat-assistant, email-classifier)
+- Source file: where the prompt text lives
+- Code file to refactor: the Python/JS file that USES the prompt (for asset files like .txt/.yaml/.md, this is the file that reads/loads the asset — NOT the asset file itself)
+- Type: chat (used as a message in a chat API) or text (used as a plain string)
+- Variables: values interpolated into the prompt, converted to {{var}} syntax:
+f-string {var} → {{var}}
+.format(var=...) → {{var}}
+${var} → {{var}}
+String concatenation + var + → {{var}}
+YAML {var} → {{var}}
+- Prompt content: the actual text to upload, with variables converted to {{var}} syntax
+
+Search for these patterns:
+
+| Framework | Look for |
+|-----------|----------|
+| OpenAI | `messages=[{"role": "system", "content": "..."}]` |
+| Anthropic | `system="..."` |
+| LangChain | `ChatPromptTemplate`, `SystemMessage` |
+| Vercel AI | `system: "..."`, `prompt: "..."` |
+| Raw | Multi-line strings near LLM calls |
+
+## Step 2: Check Templating Compatibility
+
+**CRITICAL:** Langfuse only supports simple `{{variable}}` substitution. No conditionals, loops, or filters.
+
+| Template Feature | Langfuse Native | Action |
+|------------------|-----------------|--------|
+| `{{variable}}` | ✅ | Direct migration |
+| `{var}` / `${var}` | ⚠️ | Convert to `{{var}}` |
+| `{% if %}` / `{% for %}` | ❌ | Move logic to code |
+| `{{ var \| filter }}` | ❌ | Apply filter in code |
+
+**CRITICAL — Variable syntax:** Langfuse uses DOUBLE curly braces for variables: `{{var}}`. When uploading prompt content, you MUST convert every single-brace `{var}` from the original code to double-brace `{{var}}`. Never upload `{var}` — it must be `{{var}}`.
+
+### Decision Tree
+
+```
+Contains {% if %}, {% for %}, or filters?
+├─ No → Direct migration
+└─ Yes → Choose:
+    ├─ Option A (RECOMMENDED): Move logic to code, pass pre-computed values
+    └─ Option B: Store raw template, compile client-side with Jinja2
+        └─ ⚠️ Loses: Playground preview, UI experiments
+```
+
+### Simplifying Complex Templates
+
+**Conditionals** → Pre-compute in code:
+```python
+# Instead of {% if user.is_premium %}...{% endif %} in prompt
+# Use {{tier_message}} and compute value in code before compile()
+```
+
+**Loops** → Pre-format in code:
+```python
+# Instead of {% for tool in tools %}...{% endfor %} in prompt
+# Use {{tools_list}} and format the list in code before compile()
+```
+
+For external templating details, fetch: https://langfuse.com/faq/all/using-external-templating-libraries
+
+## Step 3: Propose Structure
+
+### Naming Conventions
+
+| Rule | Example | Bad |
+|------|---------|-----|
+| Lowercase, hyphenated | `chat-assistant` | `ChatAssistant_v2` |
+| Feature-based | `document-summarizer` | `prompt1` |
+| Hierarchical for related | `support/triage` | `supportTriage` |
+| Prefix subprompts with `_` | `_base-personality` | `shared-personality` |
+
+### Identify Subprompts
+
+Extract when:
+- Same text in 2+ prompts
+- Represents distinct component (personality, safety rules, format)
+- Would need to change together
+
+### Variable Extraction
+
+| Make Variable | Keep Hardcoded |
+|---------------|----------------|
+| User-specific (`{{user_name}}`) | Output format instructions |
+| Dynamic content (`{{context}}`) | Safety guardrails |
+| Per-request (`{{query}}`) | Persona/personality |
+| Environment-specific (`{{company_name}}`) | Static examples |
+
+## Step 4: Present Plan to User
+
+Format:
+```
+Found N prompts across M files:
+
+src/chat.py:
+  - System prompt (47 lines) → 'chat-assistant'
+
+src/support/triage.py:
+  - Triage prompt (34 lines) → 'support/triage'
+    ⚠️ Contains {% if %} - will simplify
+
+Subprompts to extract:
+  - '_base-personality' - used by: chat-assistant, support/triage
+
+Variables to add:
+  - {{user_name}} - hardcoded in 2 prompts
+
+Proceed?
+```
+
+## Step 5: Create Prompts in Langfuse
+
+Use `langfuse.create_prompt()` with:
+- `name`: Your chosen name
+- `prompt`: Template text (or message array for chat type)
+- `type`: `"text"` or `"chat"`
+- `labels`: `["production"]` (they're already live)
+- `config`: Optional model settings
+
+**Labeling strategy:**
+- `production` → All migrated prompts
+- `staging` → Add later for testing
+- `latest` → Auto-applied by Langfuse
+
+For full API: fetch https://langfuse.com/docs/prompts/get-started
+
+## Step 6: Refactor Code
+
+Replace hardcoded prompts with:
+
+```python
+prompt = langfuse.get_prompt("name", label="production")
+messages = prompt.compile(var1=value1, var2=value2)
+```
+
+**Key points:**
+- Always use `label="production"` (not `latest`) for stability
+- Call `.compile()` to substitute variables
+- For chat prompts, result is message array ready for API
+
+For SDK examples (Python/JS/TS): fetch https://langfuse.com/docs/prompts/get-started
+
+## Step 7: Link Prompts to Traces
+
+If codebase uses Langfuse tracing, link prompts so you can see which version produced each response.
+
+### Detect Existing Tracing
+
+Look for:
+- `@observe()` decorators
+- `langfuse.trace()` calls
+- `from langfuse.openai import openai` (instrumented client)
+
+### Link Methods
+
+| Setup | How to Link |
+|-------|-------------|
+| `@observe()` decorator | `langfuse_context.update_current_observation(prompt=prompt)` |
+| Manual tracing | `trace.generation(prompt=prompt, ...)` |
+| OpenAI integration | `openai.chat.completions.create(..., langfuse_prompt=prompt)` |
+
+### Verify in UI
+
+1. Go to **Traces** → select a trace
+2. Click on **Generation**
+3. Check **Prompt** field shows name and version
+
+For tracing details: fetch https://langfuse.com/docs/prompts/get-started#link-with-langfuse-tracing
+
+## Step 8: Verify Migration
+
+### Checklist
+
+- [ ] All prompts created with `production` label
+- [ ] Code fetches with `label="production"`
+- [ ] Variables compile without errors
+- [ ] Subprompts resolve correctly
+- [ ] Application behavior unchanged
+- [ ] Generations show linked prompt in UI (if tracing)
+
+### Common Issues
+
+| Issue | Solution |
+|-------|----------|
+| `PromptNotFoundError` | Check name spelling |
+| Variables not replaced | Use `{{var}}` not `{var}`, call `.compile()` |
+| Subprompt not resolved | Must exist with same label |
+| Old prompt cached | Restart app |
+
+## Out of Scope
+
+- Prompt engineering (writing better prompts)
+- Evaluation setup
+- A/B testing workflow
+- Non-LLM string templates

package/.jumpstart/skills/langfuse/references/sdk-upgrade.md

@@ -0,0 +1,181 @@
+---
+name: langfuse-sdk-upgrade
+description: Upgrade Langfuse SDKs from older versions to the latest. Use when migrating Python SDK v2/v3 to v4, or JS/TS SDK v3/v4 to v5.
+---
+
+# Langfuse SDK Upgrade Guide
+
+Assist users in upgrading their Langfuse SDK to the latest version. The Python and JS/TS SDKs share the same architectural changes but differ in syntax.
+
+## When to Use
+
+- User asks to upgrade/migrate their Langfuse SDK
+- User is on an older SDK version and encounters deprecated APIs
+- User wants to adopt the latest Langfuse features
+
+## Migration Docs
+
+Always fetch the latest migration guide before starting — these pages are the source of truth:
+
+- **Python (v3 → v4):** https://langfuse.com/docs/observability/sdk/upgrade-path/python-v3-to-v4
+- **JS/TS (v4 → v5):** https://langfuse.com/docs/observability/sdk/upgrade-path/js-v4-to-v5
+
+Fetch the relevant page as markdown before implementing any changes:
+
+```bash
+curl -s "https://langfuse.com/docs/observability/sdk/upgrade-path/python-v3-to-v4.md"
+curl -s "https://langfuse.com/docs/observability/sdk/upgrade-path/js-v4-to-v5.md"
+```
+
+## Upgrade Checklist
+
+Work through each item in order. Skip items that don't apply to the user's codebase.
+
+### Both SDKs
+
+- [ ] **Update the SDK package** to the latest version
+- [ ] **Audit span filtering**: Non-LLM spans (HTTP, DB, queues) no longer export by default. If the user relied on these, configure a custom `should_export_span` / `shouldExportSpan` filter
+- [ ] **Replace `update_current_trace()` / `updateActiveTrace()`**: Split into three calls:
+  - `propagate_attributes()` / `propagateAttributes()` for correlating attributes (`user_id`, `session_id`, `tags`, `metadata`, `trace_name`)
+  - `set_current_trace_io()` / `setActiveTraceIO()` for input/output (deprecated — prefer setting I/O on root observation directly)
+  - `set_current_trace_as_public()` / `setActiveTraceAsPublic()` for public flag
+- [ ] **Replace `.update_trace()` / `.updateTrace()`** on observation objects (same decomposition as above)
+- [ ] **Update API namespace references**: `observations_v_2` / `observationsV2` → `observations`, `score_v_2` / `scoreV2` → `scores`, `metrics_v_2` / `metricsV2` → `metrics`. Legacy v1 APIs moved to `api.legacy.*`
+- [ ] **Validate metadata format**: Must be `dict[str, str]` / `Record<string, string>` with values ≤200 characters
+- [ ] **Move `release` and `environment`** from code parameters to environment variables (`LANGFUSE_RELEASE`, `LANGFUSE_TRACING_ENVIRONMENT`)
+- [ ] **Enable debug logging** during migration to catch issues (`debug=True` in Python, `LANGFUSE_DEBUG="true"` in JS/TS)
+- [ ] **Test trace hierarchies** to verify no spans are unexpectedly dropped
+
+### Python-specific
+
+- [ ] **Replace `start_span()` / `start_generation()`** with `start_observation()` (use `as_type="generation"` for generations)
+- [ ] **Replace `start_as_current_span()` / `start_as_current_generation()`** with `start_as_current_observation()`
+- [ ] **Replace dataset `item.run()`** with `dataset.run_experiment(name=..., task=...)`
+- [ ] **Remove `CallbackHandler(update_trace=...)`** parameter — use `propagate_attributes()` wrapper instead
+- [ ] **Upgrade to Pydantic v2** — the SDK now requires it. Use `pydantic.v1` compatibility shim if migrating gradually
+- [ ] **Update removed types**: `TraceMetadata`, `ObservationParams` removed from `langfuse.types`. Import `MapValue`, `ModelUsage`, `PromptClient` from `langfuse.model`
+
+### JS/TS-specific
+
+- [ ] **Update LangChain `CallbackHandler`** — `traceMetadata` now requires string values; internal behavior uses `propagateAttributes()` instead of direct trace updates
+- [ ] **Update OpenAI integration** — `traceMethod` wrapper now uses `propagateAttributes()` internally; wrap entire execution in `propagateAttributes()` if relying on parent attribute inheritance
+
+## Key API Changes Reference
+
+### Correlating attributes (both SDKs)
+
+**Before:**
+```python
+# Python
+langfuse.update_current_trace(name="trace-name", user_id="user-123", session_id="session-abc", tags=["tag1"])
+```
+```typescript
+// JS/TS
+updateActiveTrace({ name: "trace-name", userId: "user-123", sessionId: "session-456", tags: ["prod"] });
+```
+
+**After:**
+```python
+# Python
+from langfuse import propagate_attributes
+
+with propagate_attributes(trace_name="trace-name", user_id="user-123", session_id="session-abc", tags=["tag1"]):
+    result = call_llm("hello")
+```
+```typescript
+// JS/TS
+import { propagateAttributes } from "langfuse";
+
+await propagateAttributes(
+  { traceName: "trace-name", userId: "user-123", sessionId: "session-456", tags: ["prod"] },
+  async () => { /* traced code */ }
+);
+```
+
+### Span/Generation creation (Python)
+
+**Before:**
+```python
+langfuse.start_span(name="x")
+langfuse.start_generation(name="x", model="gpt-4")
+```
+
+**After:**
+```python
+langfuse.start_observation(name="x")
+langfuse.start_observation(name="x", as_type="generation", model="gpt-4")
+```
+
+### Dataset experiments (Python)
+
+**Before:**
+```python
+for item in dataset.items:
+    with item.run(run_name="my-run") as span:
+        result = my_llm(item.input)
+        span.update(output=result)
+```
+
+**After:**
+```python
+def my_task(*, item, **kwargs):
+    return my_llm(item.input)
+
+dataset.run_experiment(name="my-run", task=my_task)
+```
+
+### Span filtering (both SDKs)
+
+To restore pre-upgrade "export all" behavior:
+
+```python
+# Python
+langfuse = Langfuse(should_export_span=lambda span: True)
+```
+```typescript
+// JS/TS
+const spanProcessor = new LangfuseSpanProcessor({ shouldExportSpan: () => true });
+```
+
+To extend defaults with custom scopes:
+
+```python
+# Python
+from langfuse.span_filter import is_default_export_span
+
+langfuse = Langfuse(
+    should_export_span=lambda span: (
+        is_default_export_span(span)
+        or span.instrumentation_scope.name.startswith("my_framework")
+    )
+)
+```
+```typescript
+// JS/TS
+import { isDefaultExportSpan } from "@langfuse/otel";
+
+shouldExportSpan: ({ otelSpan }) =>
+  isDefaultExportSpan(otelSpan) || otelSpan.instrumentationScope.name.startsWith("my_framework")
+```
+
+## Common Pitfalls
+
+| Pitfall | Impact | Fix |
+| --- | --- | --- |
+| Dropping intermediate spans via filtering | Breaks trace trees — child spans become orphaned | Use `is_default_export_span` as base and only add/remove specific scopes |
+| Metadata with non-string values | Values silently coerced or dropped | Ensure all metadata values are strings ≤200 characters |
+| Setting attributes outside `propagate_attributes()` callback | Attributes don't attach to observations | Wrap all traced code inside the callback |
+| Using deprecated `set_current_trace_io()` for new code | Will be removed in future versions | Set input/output directly on the root observation |
+| Forgetting Pydantic v2 upgrade (Python) | Import errors or runtime failures | Upgrade Pydantic or use `pydantic.v1` shim |
+| `release`/`environment` still passed as parameters | Silently ignored | Use `LANGFUSE_RELEASE` and `LANGFUSE_TRACING_ENVIRONMENT` env vars |
+| LangChain/OpenAI attribute propagation direction changed | Attributes propagate downward only, not upward to parent traces | Wrap outer call in `propagate_attributes()` |
+
+## Best Practices
+
+1. **Always fetch the migration docs first** — they are the canonical source and may have been updated since this guide was written
+2. **Enable debug logging during migration** to surface dropped spans and trace hierarchy issues
+3. **Use `propagate_attributes()` as the primary mechanism** for setting trace-level correlating attributes
+4. **Set input/output on root observations directly** rather than using deprecated trace-level setters
+5. **Compose custom span filters** with `is_default_export_span` / `isDefaultExportSpan` to extend defaults rather than replacing them entirely
+6. **Test thoroughly** — run the application with debug logging, check the Langfuse UI for missing or orphaned spans, verify metadata appears correctly
+7. **Migrate incrementally** — upgrade the SDK first, fix breaking changes, then adopt new patterns

package/.jumpstart/skills/langfuse/references/skill-feedback.md

@@ -0,0 +1,52 @@
+---
+name: langfuse-skill-feedback
+description: Submit feedback about the Langfuse skill to its maintainers via GitHub Discussions. Use when the user indicates the skill gave incorrect guidance, is missing information, or could be improved.
+---
+
+# Skill Feedback
+
+Follow these steps exactly:
+
+1. **Ask permission**: Ask the user if they'd like you to submit feedback to the skill maintainers. Make it clear this is about the skill (the agent instructions), not about Langfuse the product. If they decline, move on.
+2. **Draft feedback**: Write the feedback using the form structure below. Present the draft to the user and ask if they'd like to change anything before submitting.
+3. **Submit**: Once approved, submit via `gh` CLI as described below. Share the resulting discussion URL with the user.
+
+## Feedback Form Structure
+
+Draft the feedback using these two fields:
+
+**Describe your idea or feedback** (required)
+A clear description of what went wrong or what could be improved. Include:
+- What the user was trying to do
+- What the skill did vs what was expected
+- Any specific instructions that were incorrect or missing
+
+**What would the ideal outcome look like?** (optional)
+What the correct behavior or guidance should be.
+
+Format the body as markdown with the two field labels as headings.
+
+## Submitting
+
+Create a GitHub Discussion on the `langfuse/skills` repository using the GraphQL API:
+
+```bash
+gh api graphql -f query='
+mutation($repoId: ID!, $categoryId: ID!, $title: String!, $body: String!) {
+  createDiscussion(input: {repositoryId: $repoId, categoryId: $categoryId, title: $title, body: $body}) {
+    discussion { url }
+  }
+}' \
+  -f repoId="$(gh api graphql -f query='{ repository(owner: "langfuse", name: "skills") { id } }' --jq '.data.repository.id')" \
+  -f categoryId="$(gh api graphql -f query='{ repository(owner: "langfuse", name: "skills") { discussionCategories(first: 10) { nodes { id name } } } }' --jq '.data.repository.discussionCategories.nodes[] | select(.name == "Ideas & Improvements") | .id')" \
+  -f title="<concise title>" \
+  -f body="<formatted feedback>"
+```
+
+If the `gh` CLI is not authenticated or the request fails, give the user this link to create the discussion manually:
+
+```
+https://github.com/langfuse/skills/discussions/new?category=ideas-improvements
+```
+
+After submission, share the discussion URL with the user.

package/.jumpstart/skills/langfuse/references/user-feedback.md

@@ -0,0 +1,88 @@
+---
+name: langfuse-user-feedback
+description: Wires up user feedback (thumbs up/down, ratings, comments) from an application's frontend to Langfuse scores. Use when user wants to capture end-user feedback, add ratings to traces, or connect user complaints to Langfuse.
+---
+
+# User Feedback
+
+Tracing must already be set up — feedback is stored as scores on traces.
+
+Docs: https://langfuse.com/docs/observability/features/user-feedback
+
+## Workflow
+
+### 1. Determine What Feedback to Capture
+
+If the user has asked for something specific, go with that. Otherwise, look at the application and **present a few UX options** for how feedback could work, then ask the user which they prefer before implementing.
+
+Common UX patterns to suggest:
+
+| UX Pattern | Best for | How it works |
+|------------|----------|--------------|
+| Thumbs up/down | Chat apps, Q&A | Simple binary buttons next to each response |
+| Star rating (1–5) | Content generation, summaries | Star row or dropdown after each output |
+| "Was this helpful?" banner | Search, documentation assistants | Single yes/no prompt at the bottom of a response |
+| Regenerate / copy tracking | Any app with these actions | Implicit — log when users retry (negative signal) or copy output (positive signal) |
+| Free-text comment | Complex outputs, internal tools | Optional text field alongside a rating |
+| Report button | Any user-facing app | Flag icon to report bad/harmful responses |
+
+This table is not exhaustive — if the application suggests a different feedback pattern that fits better, propose that instead. Present 2–3 options that match the application's use case and ask the user which approach they'd like. This decision shapes everything downstream (score names, data types, frontend components), so it's important to align early.
+
+Feedback can be **explicit** (user rates via thumbs, stars, etc.) or **implicit** (derived from behavior like copying output, retrying, or escalating to support). Both are stored as scores. Explicit feedback requires the trace ID to reach the frontend; implicit feedback is logged server-side where the event already happens.
+
+### 2. Choose Score Names
+
+Name reflects the signal source, not what you hope it measures (e.g., `user-thumbs` not `response-quality` — a thumbs down doesn't tell you *what* was wrong). Avoid generic names like `feedback` or `score`.
+
+Rules:
+- Lowercase with hyphens
+- One consistent name per feedback type across the entire app
+- If capturing multiple signals, each gets its own distinct name
+
+### 3. Implement Score Creation
+
+**For implicit feedback (server-side):** Use `langfuse.create_score()` / `langfuse.score.create()` wherever the event is already handled in application code. Fetch SDK docs for current API: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk
+
+**For explicit feedback (frontend):** Use `LangfuseWeb` in the browser. It uses the public key only — no secret key exposed.
+
+```typescript
+import { LangfuseWeb } from "langfuse";
+
+const langfuse = new LangfuseWeb({
+  publicKey: process.env.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY!,
+  baseUrl: process.env.NEXT_PUBLIC_LANGFUSE_HOST,
+});
+
+langfuse.score({
+  traceId,
+  name: "user-thumbs",
+  value: 1, // 1 = positive, 0 = negative
+  dataType: "BOOLEAN",
+  comment: optionalUserComment,
+});
+```
+
+The trace ID must be available in the frontend for this to work. For Vercel AI SDK, the non-obvious pattern is using `generateMessageId`:
+
+```typescript
+import { getActiveTraceId } from "@langfuse/tracing";
+
+// Inside route handler wrapped with observe()
+return result.toUIMessageStreamResponse({
+  generateMessageId: () => getActiveTraceId() || crypto.randomUUID(),
+});
+```
+
+### 4. Verify
+
+Trigger a feedback action and check the trace's Scores tab in Langfuse. Confirm the score name, value, and data type are correct.
+
+Point users to what they can do with feedback data: filter traces by low scores, use score analytics for trends, build annotation queues for team review.
+
+## Common Mistakes
+
+| Mistake | Problem | Fix |
+|---------|---------|-----|
+| Secret key in frontend code | Security risk | Use `LangfuseWeb` with public key only |
+| Missing `dataType` on boolean scores | Value `1` inferred as `NUMERIC` | Always pass `dataType: "BOOLEAN"` explicitly |
+| Inconsistent score names across the app | Can't aggregate or filter reliably | Pick one name per feedback type, use it everywhere |