quiver-cli 0.1.0

package/template/.agents/skills/integrations/posthog/references/llm-analytics-anthropic.md

@@ -0,0 +1,200 @@
+# Anthropic LLM analytics installation - Docs
+
+1.  1
+
+    ## Install the PostHog SDK
+
+    Required
+
+    Setting up analytics starts with installing the PostHog SDK for your language. LLM analytics works best with our Python and Node SDKs.
+
+    PostHog AI
+
+    ### Python
+
+    ```bash
+    pip install posthog
+    ```
+
+    ### Node
+
+    ```bash
+    npm install @posthog/ai posthog-node
+    ```
+
+2.  2
+
+    ## Install the Anthropic SDK
+
+    Required
+
+    Install the Anthropic SDK. The PostHog SDK instruments your LLM calls by wrapping the Anthropic client. The PostHog SDK **does not** proxy your calls.
+
+    PostHog AI
+
+    ### Python
+
+    ```bash
+    pip install anthropic
+    ```
+
+    ### Node
+
+    ```bash
+    npm install @anthropic-ai/sdk
+    ```
+
+    **Proxy note**
+
+    These SDKs **do not** proxy your calls. They only fire off an async call to PostHog in the background to send the data. You can also use LLM analytics with other SDKs or our API, but you will need to capture the data in the right format. See the schema in the [manual capture section](/docs/llm-analytics/installation/manual-capture.md) for more details.
+
+3.  3
+
+    ## Initialize PostHog and the Anthropic wrapper
+
+    Required
+
+    Initialize PostHog with your project token and host from [your project settings](https://app.posthog.com/settings/project), then pass it to our Anthropic wrapper.
+
+    PostHog AI
+
+    ### Python
+
+    ```python
+    from posthog.ai.anthropic import Anthropic
+    from posthog import Posthog
+    posthog = Posthog(
+        "<ph_project_token>",
+        host="https://us.i.posthog.com"
+    )
+    client = Anthropic(
+        api_key="sk-ant-api...", # Replace with your Anthropic API key
+        posthog_client=posthog # This is an optional parameter. If it is not provided, a default client will be used.
+    )
+    ```
+
+    ### Node
+
+    ```typescript
+    import { Anthropic } from '@posthog/ai'
+    import { PostHog } from 'posthog-node'
+    const phClient = new PostHog(
+      '<ph_project_token>',
+      { host: 'https://us.i.posthog.com' }
+    )
+    const client = new Anthropic({
+      apiKey: 'sk-ant-api...', // Replace with your Anthropic API key
+      posthog: phClient
+    })
+    ```
+
+    > **Note:** This also works with the `AsyncAnthropic` client as well as `AnthropicBedrock`, `AnthropicVertex`, and the async versions of those.
+
+4.  4
+
+    ## Call Anthropic LLMs
+
+    Required
+
+    Now, when you use the Anthropic SDK to call LLMs, PostHog automatically captures an `$ai_generation` event. You can enrich the event with additional data such as the trace ID, distinct ID, custom properties, groups, and privacy mode options.
+
+    PostHog AI
+
+    ### Python
+
+    ```python
+    response = client.messages.create(
+        model="claude-3-opus-20240229",
+        messages=[
+            {
+                "role": "user",
+                "content": "Tell me a fun fact about hedgehogs"
+            }
+        ],
+        posthog_distinct_id="user_123", # optional
+        posthog_trace_id="trace_123", # optional
+        posthog_properties={"conversation_id": "abc123", "paid": True}, # optional
+        posthog_groups={"company": "company_id_in_your_db"},  # optional
+        posthog_privacy_mode=False # optional
+    )
+    print(response.content[0].text)
+    ```
+
+    ### Node
+
+    ```typescript
+    const response = await client.messages.create({
+      model: "claude-3-5-sonnet-latest",
+      messages: [
+        {
+          role: "user",
+          content: "Tell me a fun fact about hedgehogs"
+        }
+      ],
+      posthogDistinctId: "user_123", // optional
+      posthogTraceId: "trace_123", // optional
+      posthogProperties: { conversationId: "abc123", paid: true }, // optional
+      posthogGroups: { company: "company_id_in_your_db" }, // optional
+      posthogPrivacyMode: false // optional
+    })
+    console.log(response.content[0].text)
+    phClient.shutdown()
+    ```
+
+    > **Notes:**
+    >
+    > -   This also works when message streams are used (e.g. `stream=True` or `client.messages.stream(...)`).
+    > -   If you want to capture LLM events anonymously, **don't** pass a distinct ID to the request.
+    >
+    > See our docs on [anonymous vs identified events](/docs/data/anonymous-vs-identified-events.md) to learn more.
+
+    You can expect captured `$ai_generation` events to have the following properties:
+
+    | Property | Description |
+    | --- | --- |
+    | $ai_model | The specific model, like gpt-5-mini or claude-4-sonnet |
+    | $ai_latency | The latency of the LLM call in seconds |
+    | $ai_time_to_first_token | Time to first token in seconds (streaming only) |
+    | $ai_tools | Tools and functions available to the LLM |
+    | $ai_input | List of messages sent to the LLM |
+    | $ai_input_tokens | The number of tokens in the input (often found in response.usage) |
+    | $ai_output_choices | List of response choices from the LLM |
+    | $ai_output_tokens | The number of tokens in the output (often found in response.usage) |
+    | $ai_total_cost_usd | The total cost in USD (input + output) |
+    | [[...]](/docs/llm-analytics/generations.md#event-properties) | See [full list](/docs/llm-analytics/generations.md#event-properties) of properties |
+
+5.  ## Verify traces and generations
+
+    Recommended
+
+    *Confirm LLM events are being sent to PostHog*
+
+    Let's make sure LLM events are being captured and sent to PostHog. Under **LLM analytics**, you should see rows of data appear in the **Traces** and **Generations** tabs.
+
+    ![LLM generations in PostHog](https://res.cloudinary.com/dmukukwp6/image/upload/SCR_20250807_syne_ecd0801880.png)![LLM generations in PostHog](https://res.cloudinary.com/dmukukwp6/image/upload/SCR_20250807_syjm_5baab36590.png)
+
+    [Check for LLM events in PostHog](https://app.posthog.com/llm-analytics/generations)
+
+6.  5
+
+    ## Next steps
+
+    Recommended
+
+    Now that you're capturing AI conversations, continue with the resources below to learn what else LLM Analytics enables within the PostHog platform.
+
+    | Resource | Description |
+    | --- | --- |
+    | [Basics](/docs/llm-analytics/basics.md) | Learn the basics of how LLM calls become events in PostHog. |
+    | [Generations](/docs/llm-analytics/generations.md) | Read about the $ai_generation event and its properties. |
+    | [Traces](/docs/llm-analytics/traces.md) | Explore the trace hierarchy and how to use it to debug LLM calls. |
+    | [Spans](/docs/llm-analytics/spans.md) | Review spans and their role in representing individual operations. |
+    | [Anaylze LLM performance](/docs/llm-analytics/dashboard.md) | Learn how to create dashboards to analyze LLM performance. |
+
+### Community questions
+
+Ask a question
+
+### Was this page useful?
+
+HelpfulCould be better

package/template/.agents/skills/integrations/posthog/references/llm-analytics-basics.md

@@ -0,0 +1,62 @@
+# LLM Analytics basics - Docs
+
+This page covers how your LLM calls become analytics in PostHog and defines key concepts for LLM Analytics.
+
+## How LLM calls become events
+
+PostHog's LLM Analytics works by wrapping your existing LLM provider's SDK to capture requests and responses. Your API calls still go directly to your provider, but the wrapper extracts metadata from each call and sends it to PostHog as an event.
+
+graph LR Call\["LLM call"\] --> Wrapper\["Wrapped SDK"\] Wrapper --> Provider\["LLM provider API"\] Wrapper --> Event\["Event"\] Event --> PostHog\["PostHog"\]
+
+Events sent to PostHog are called **generations**. A generation represents a single LLM call. For example, when you send a prompt to Claude and get a response back, that's one generation.
+
+Generations are represented using the event name `$ai_generation`. Each generation captures the model, provider, input, output, token counts, latency, and cost.
+
+## Generation properties
+
+### Tokens and costs
+
+**Tokens** are the units LLMs use to process text. LLM providers charge based on token usage:
+
+-   **Input tokens** are tokens used in the message you send to an LLM
+-   **Output tokens** are tokens used in the message you receive from an LLM
+
+PostHog automatically calculates costs by matching your model and provider against pricing data. We use OpenRouter's pricing as our primary source, with fallback to manually maintained pricing for additional models.
+
+You can also [set custom pricing](/docs/llm-analytics/calculating-costs.md) if you have negotiated rates or use unsupported models.
+
+### Message roles
+
+When you send messages to an LLM, each message has a **role** that tells the model how to interpret it:
+
+| Role | Purpose | Example |
+| --- | --- | --- |
+| system | Instructions that define the assistant's behavior | "You are a helpful assistant that speaks like a pirate" |
+| user | Messages from the end user | "What's the weather today?" |
+| assistant | Previous model responses, used for conversation history | "Arrr, it be sunny with a chance of scurvy!" |
+
+PostHog captures the full message array with roles intact, so you can see exactly what context the model had when it generated a response.
+
+## Grouping related events
+
+Most LLM applications involve multiple calls. Traces, spans, and sessions let you see how they connect:
+
+graph TD Session\["Session"\] Trace\["Trace"\] Generation1\["Generation"\] Span\["Span"\] Generation2\["Generation"\] Session -.-> Trace Trace --> Generation1 Trace --> Span Span --> Generation2
+
+Here's a breakdown of this hierarchy:
+
+| Term | Definition | Example |
+| --- | --- | --- |
+| [Session](/docs/llm-analytics/sessions.md) | Groups multiple traces together | A user's conversation thread |
+| [Trace](/docs/llm-analytics/traces.md) | Contains generations and spans for a single request | One chatbot message and response |
+| [Span](/docs/llm-analytics/spans.md) | Tracks an operation within a trace | A retrieval step or function call |
+| [Generation](/docs/llm-analytics/generations.md) | An LLM call, tracked as $ai_generation events | Sending a prompt to Claude |
+| [Embedding](/docs/llm-analytics/embeddings.md) | Converts text into vectors | Vectorizing documents for RAG |
+
+### Community questions
+
+Ask a question
+
+### Was this page useful?
+
+HelpfulCould be better

package/template/.agents/skills/integrations/posthog/references/llm-analytics-costs.md

@@ -0,0 +1,197 @@
+# Calculating LLM costs - Docs
+
+## How are LLM costs calculated?
+
+PostHog calculates cost based on the number of input (prompt) and output (completion) tokens generated by specific AI models, plus additional cost components like per-request and per-web-search pricing when applicable.
+
+### Model matching
+
+We use OpenRouter's pricing data as our primary source to match your model and calculate costs. OpenRouter provides comprehensive pricing information for models across different providers, and we use both the `$ai_provider` and `$ai_model` properties from your events to find the exact pricing.
+
+If OpenRouter doesn't have pricing data for a specific model, we fall back to our manually maintained pricing database for additional model support.
+
+### Cost components
+
+The total cost (`$ai_total_cost_usd`) is calculated from multiple components:
+
+-   **Token-based costs**: Input (prompt) and output (completion) tokens
+-   **Cache costs**: Cached token reads and writes (when applicable)
+-   **Request-based costs**: Some models charge per request in addition to token costs
+-   **Web search costs**: Some models charge per web search performed
+
+For cached LLM responses, our pricing models include cached token pricing which we automatically apply.
+
+We also take into account the reasoning / thinking tokens for models that support it.
+
+### Cache token accounting
+
+Different LLM providers report cache tokens differently:
+
+-   **Exclusive counting** (Anthropic/Claude) - Cache tokens are separate from `$ai_input_tokens`. For example, if you have 100 input tokens and 50 cached tokens, `$ai_input_tokens` is 100.
+-   **Inclusive counting** (OpenAI and most others) - Cache tokens are included in `$ai_input_tokens`. Using the same example, `$ai_input_tokens` is 150.
+
+PostHog auto-detects which counting style to use based on the `$ai_provider` and `$ai_model` properties. For Anthropic and Claude models, PostHog assumes exclusive counting. For all other providers, it assumes inclusive counting.
+
+#### Overriding auto-detection
+
+If you're manually capturing events or using a framework wrapper that normalizes token counts differently, override the auto-detection by setting the `$ai_cache_reporting_exclusive` property:
+
+-   `$ai_cache_reporting_exclusive: true` - Cache tokens are separate from input tokens (Anthropic-style)
+-   `$ai_cache_reporting_exclusive: false` - Input tokens already include cache tokens
+
+When not set, PostHog resolves this automatically and writes the resolved value back to the event for downstream consumers.
+
+## Setting custom pricing
+
+You can override PostHog's automatic cost calculation by providing custom pricing for your LLM models. This is useful when:
+
+-   You have negotiated custom pricing with your LLM provider
+-   You're using a model that PostHog doesn't support yet
+-   PostHog's automatic pricing doesn't match your specific use case
+
+### Option 1: Custom price per token
+
+If you know your pricing per token, you can set the following [custom properties](/docs/llm-analytics/custom-properties.md) when calling your LLM:
+
+-   `$ai_input_token_price` (required): Price per input/prompt token
+-   `$ai_output_token_price` (required): Price per output/completion token
+-   `$ai_cache_read_token_price` (optional): Price per cached token read
+-   `$ai_cache_write_token_price` (optional): Price per cached token write
+-   `$ai_request_price` (optional): Price per request (for models that charge per request)
+-   `$ai_web_search_price` (optional): Price per web search (for models that charge per search)
+
+**Important:** Token prices should be per individual token, not per million tokens. For example, if your provider charges $0.03 per 1M tokens, you would set `$ai_input_token_price: 0.00000003` (0.03 / 1,000,000).
+
+PostHog AI
+
+### JavaScript
+
+```javascript
+import { OpenAI } from '@posthog/ai'
+import { PostHog } from 'posthog-node'
+const phClient = new PostHog(
+  '<ph_project_token>',
+  { host: 'https://us.i.posthog.com' }
+)
+const openai = new OpenAI({
+  apiKey: process.env.OPENAI_API_KEY,
+  posthog: phClient
+})
+const response = await openai.responses.create({
+  model: 'my-custom-model',
+  messages: [{ role: 'user', content: 'Hello' }],
+  posthogProperties: {
+    $ai_input_token_price: 0.00000003,   // $0.03 per 1M tokens = $0.00000003 per token
+    $ai_output_token_price: 0.00000006,  // $0.06 per 1M tokens = $0.00000006 per token
+    // Optional: cache pricing
+    $ai_cache_read_token_price: 0.000000015,
+    $ai_cache_write_token_price: 0.0000000375
+  }
+})
+```
+
+### Python
+
+```python
+from posthog.ai.openai import OpenAI
+from posthog import Posthog
+posthog = Posthog(
+    "<ph_project_token>",
+    host="https://us.i.posthog.com"
+)
+client = OpenAI(
+    api_key="sk-...",
+    posthog_client=posthog
+)
+response = client.responses.create(
+    model="my-custom-model",
+    messages=[{"role": "user", "content": "Hello"}],
+    posthog_properties={
+        "$ai_input_token_price": 0.00000003,   # $0.03 per 1M tokens = $0.00000003 per token
+        "$ai_output_token_price": 0.00000006,  # $0.06 per 1M tokens = $0.00000006 per token
+        # Optional: cache pricing
+        "$ai_cache_read_token_price": 0.000000015,
+        "$ai_cache_write_token_price": 0.0000000375
+    }
+)
+```
+
+Both `$ai_input_token_price` and `$ai_output_token_price` must be provided for custom pricing to take effect. PostHog will then calculate the total cost based on the token counts and your custom prices.
+
+### Option 2: Pre-calculated costs
+
+If you're [manually capturing](/docs/llm-analytics/installation/manual-capture.md) LLM events and have already calculated the total costs yourself, you can send them directly:
+
+-   `$ai_input_cost_usd`: Total cost for input/prompt tokens in USD
+-   `$ai_output_cost_usd`: Total cost for output/completion tokens in USD
+-   `$ai_request_cost_usd`: Total cost for requests in USD
+-   `$ai_web_search_cost_usd`: Total cost for web searches in USD
+
+PostHog AI
+
+### JavaScript
+
+```javascript
+// After making your LLM call and calculating costs
+posthog.capture('$ai_generation', {
+  $ai_trace_id: traceId,
+  $ai_model: 'my-custom-model',
+  $ai_provider: 'my-provider',
+  $ai_input_tokens: inputTokens,
+  $ai_output_tokens: outputTokens,
+  $ai_input_cost_usd: 0.0042,
+  $ai_output_cost_usd: 0.0028
+  // ... other required properties
+})
+```
+
+### Python
+
+```python
+# After making your LLM call and calculating costs
+posthog.capture(
+    event='$ai_generation',
+    properties={
+        '$ai_trace_id': trace_id,
+        '$ai_model': 'my-custom-model',
+        '$ai_provider': 'my-provider',
+        '$ai_input_tokens': input_tokens,
+        '$ai_output_tokens': output_tokens,
+        '$ai_input_cost_usd': 0.0042,
+        '$ai_output_cost_usd': 0.0028
+        # ... other required properties
+    }
+)
+```
+
+PostHog will use these values directly without any additional calculation.
+
+### Precedence
+
+Cost calculation follows this precedence order:
+
+1.  **Pre-calculated costs** (`$ai_input_cost_usd`, `$ai_output_cost_usd`, `$ai_request_cost_usd`, `$ai_web_search_cost_usd`): These values are used directly without any additional calculation
+2.  **Custom pricing** (`$ai_input_token_price`, `$ai_output_token_price`, `$ai_request_price`, `$ai_web_search_price`): PostHog calculates costs from token counts and usage metrics using your custom prices
+3.  **Automatic model matching**: PostHog matches your model using OpenRouter's pricing data first, then falls back to manually maintained pricing if needed
+
+## Cost calculation metadata
+
+When PostHog calculates costs automatically, it sets the following metadata properties on the event to help you understand how costs were determined:
+
+| Property | Description | Examples |
+| --- | --- | --- |
+| $ai_model_cost_used | The model identifier used for cost calculation. May differ from the requested model when a variant or alias is resolved. | openai/gpt-4o-mini |
+| $ai_cost_model_source | Where the cost data for this model was sourced from. | openrouter, manual, custom, passthrough |
+| $ai_cost_model_provider | The provider used to look up the cost for this model. | openai, anthropic, custom |
+
+These properties are useful for debugging cost discrepancies or understanding which pricing was applied when using model aliases or custom configurations.
+
+You can find the code for this on [GitHub](https://github.com/PostHog/posthog/tree/master/plugin-server/src/ingestion/ai-costs).
+
+### Community questions
+
+Ask a question
+
+### Was this page useful?
+
+HelpfulCould be better