variably-sdk 2.6.0__tar.gz → 2.7.0__tar.gz

{variably_sdk-2.6.0 → variably_sdk-2.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: variably-sdk
-Version: 2.6.0
+Version: 2.7.0
 Summary: Official Python SDK for Variably feature flags, LLM experimentation, and prompt optimization platform
 Author: Variably
 Author-email: Variably <support@variably.com>
@@ -45,7 +45,7 @@ Dynamic: requires-python
 
 # Variably Python SDK
 
-Official Python SDK for Variably — feature flags, LLM experimentation, and prompt optimization.
+Official Python SDK for Variably — LLM evaluation, experimentation, and prompt optimization.
 
 ## Installation
 
@@ -53,52 +53,158 @@ Official Python SDK for Variably — feature flags, LLM experimentation, and pro
 pip install variably-sdk
 ```
 
-## Quick Start
+For Docker/Kubernetes deployments, add to your `requirements.txt`:
+
+```
+variably-sdk>=2.6.1
+```
+## Quick Start — Observe Mode
+
+Add **one line** to your existing LLM app and get **multi-dimension evaluation** across 40+ metrics in 6 categories: Quality, Safety, Semantic, Grounding, Coherence, and Advanced.
+
+No experiment setup. No prompt migration. Just log and see scores.
+
+### 1. Set your environment variables
+
+```bash
+VARIABLY_API_KEY=vbl_your_key_here
+VARIABLY_BASE_URL=https://api.variably.tech
+```
+
+### 2. Add one line after your LLM call
 
 ```python
-from variably import VariablyClient
+from variably import observe
 
-# Initialize the client
-client = VariablyClient({
-    "api_key": "your-api-key",
-    "base_url": "https://api.variably.com",  # optional, defaults to localhost:8080
-    "environment": "production"  # optional
-})
+# Your existing code (unchanged)
+response = your_llm_call(user_query)
 
-# Evaluate a boolean feature flag
-user_context = {
-    "user_id": "user-123",
-    "email": "user@example.com",
-    "country": "US"
-}
+# Add this line:
+observe(prompt=user_query, response=response)
+```
+
+### Auto-extract tokens & model from provider response
 
-is_feature_enabled = client.evaluate_flag_bool(
-    "new-checkout-flow",
-    False,  # default value
-    user_context
+```python
+# OpenAI
+from openai import OpenAI
+from variably import observe
+
+client = OpenAI()
+completion = client.chat.completions.create(
+    model="gpt-4o",
+    messages=[{"role": "user", "content": user_query}],
 )
 
-if is_feature_enabled:
-    # Show new checkout flow
-    pass
+observe(
+    prompt=user_query,
+    response=completion.choices[0].message.content,
+    provider_response=completion,  # auto-extracts model, tokens
+)
+```
 
-# Evaluate a feature gate
-has_access = client.evaluate_gate("premium-features", user_context)
+```python
+# Anthropic
+import anthropic
+from variably import observe
+
+client = anthropic.Anthropic()
+message = client.messages.create(
+    model="claude-sonnet-4-20250514",
+    max_tokens=1024,
+    messages=[{"role": "user", "content": user_query}],
+)
 
-# Track events
-client.track({
-    "name": "button_clicked",
-    "user_id": "user-123",
-    "properties": {
-        "button_name": "checkout",
-        "page": "product-detail"
-    }
-})
+observe(
+    prompt=user_query,
+    response=message.content[0].text,
+    provider_response=message,  # auto-extracts model, tokens
+)
+```
+
+### RAG applications — grounding & hallucination scoring
 
-# Clean up resources
-client.close()
+```python
+observe(
+    prompt=user_query,
+    response=llm_answer,
+    provider_response=completion,
+    reference_materials=[
+        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
+        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
+    ],
+    retrieval_query=user_query,
+)
 ```
 
+### Multi-turn chat — coherence scoring
+
+```python
+observe(
+    prompt=latest_user_message,
+    response=llm_answer,
+    provider_response=completion,
+    conversation_history=[
+        {"role": "user", "content": "What is diabetes?"},
+        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
+        {"role": "user", "content": "What are the symptoms?"},
+    ],
+    session_id="conv-123",
+)
+```
+
+### RAG applications — grounding & hallucination scoring
+
+```python
+observe(
+    prompt=user_query,
+    response=llm_answer,
+    provider_response=completion,
+    reference_materials=[
+        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
+        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
+    ],
+    retrieval_query=user_query,
+)
+```
+
+### Multi-turn chat — coherence scoring
+
+```python
+observe(
+    prompt=latest_user_message,
+    response=llm_answer,
+    provider_response=completion,
+    conversation_history=[
+        {"role": "user", "content": "What is diabetes?"},
+        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
+        {"role": "user", "content": "What are the symptoms?"},
+    ],
+    session_id="conv-123",
+)
+```
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `prompt` | str | Yes | The user's input / question |
+| `response` | str | Yes | The LLM's generated response |
+| `provider_response` | object | No | Raw OpenAI/Anthropic/Google response — auto-extracts model, tokens |
+| `model` | str | No | Model name (auto-extracted if provider_response given) |
+| `provider` | str | No | "openai", "anthropic", etc. (auto-detected) |
+| `latency_ms` | int | No | Response generation time in milliseconds |
+| `prompt_tokens` | int | No | Input token count (auto-extracted if provider_response given) |
+| `completion_tokens` | int | No | Output token count (auto-extracted) |
+| `cost` | float | No | Cost in USD |
+| `reference_materials` | list[dict] | No | RAG chunks: `[{"id", "content", "source"}]` — enables grounding scoring |
+| `retrieval_query` | str | No | Query sent to retriever — enables retrieval quality scoring |
+| `conversation_history` | list[dict] | No | Prior turns: `[{"role", "content"}]` — enables coherence scoring |
+| `tags` | list[str] | No | Grouping labels, e.g. `["production", "rag"]` |
+| `user_id` | str | No | Your user's ID |
+| `session_id` | str | No | Conversation session ID (groups multi-turn) |
+| `metadata` | dict | No | Any extra key-value data |
+
+---
+
 ## Prompt Experimentation
 
 Variably provides two modes for LLM prompt experimentation:
@@ -293,7 +399,7 @@ result = client.submit_response(
 | `workflow_history[].content` | `str` | Message content |
 | `retrieval_query` | `str` (optional) | The rewritten query sent to the retriever |
 
-See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
+See [Context-Aware RAG Evaluation](../../docs/concepts/LLM%20Evaluation/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
 
 #### Integration with LangGraph / FastAPI streaming
 

{variably_sdk-2.6.0 → variably_sdk-2.7.0}/README.md

@@ -1,6 +1,6 @@
 # Variably Python SDK
 
-Official Python SDK for Variably — feature flags, LLM experimentation, and prompt optimization.
+Official Python SDK for Variably — LLM evaluation, experimentation, and prompt optimization.
 
 ## Installation
 
@@ -8,52 +8,158 @@ Official Python SDK for Variably — feature flags, LLM experimentation, and pro
 pip install variably-sdk
 ```
 
-## Quick Start
+For Docker/Kubernetes deployments, add to your `requirements.txt`:
+
+```
+variably-sdk>=2.6.1
+```
+## Quick Start — Observe Mode
+
+Add **one line** to your existing LLM app and get **multi-dimension evaluation** across 40+ metrics in 6 categories: Quality, Safety, Semantic, Grounding, Coherence, and Advanced.
+
+No experiment setup. No prompt migration. Just log and see scores.
+
+### 1. Set your environment variables
+
+```bash
+VARIABLY_API_KEY=vbl_your_key_here
+VARIABLY_BASE_URL=https://api.variably.tech
+```
+
+### 2. Add one line after your LLM call
 
 ```python
-from variably import VariablyClient
+from variably import observe
 
-# Initialize the client
-client = VariablyClient({
-    "api_key": "your-api-key",
-    "base_url": "https://api.variably.com",  # optional, defaults to localhost:8080
-    "environment": "production"  # optional
-})
+# Your existing code (unchanged)
+response = your_llm_call(user_query)
 
-# Evaluate a boolean feature flag
-user_context = {
-    "user_id": "user-123",
-    "email": "user@example.com",
-    "country": "US"
-}
+# Add this line:
+observe(prompt=user_query, response=response)
+```
+
+### Auto-extract tokens & model from provider response
 
-is_feature_enabled = client.evaluate_flag_bool(
-    "new-checkout-flow",
-    False,  # default value
-    user_context
+```python
+# OpenAI
+from openai import OpenAI
+from variably import observe
+
+client = OpenAI()
+completion = client.chat.completions.create(
+    model="gpt-4o",
+    messages=[{"role": "user", "content": user_query}],
 )
 
-if is_feature_enabled:
-    # Show new checkout flow
-    pass
+observe(
+    prompt=user_query,
+    response=completion.choices[0].message.content,
+    provider_response=completion,  # auto-extracts model, tokens
+)
+```
 
-# Evaluate a feature gate
-has_access = client.evaluate_gate("premium-features", user_context)
+```python
+# Anthropic
+import anthropic
+from variably import observe
+
+client = anthropic.Anthropic()
+message = client.messages.create(
+    model="claude-sonnet-4-20250514",
+    max_tokens=1024,
+    messages=[{"role": "user", "content": user_query}],
+)
 
-# Track events
-client.track({
-    "name": "button_clicked",
-    "user_id": "user-123",
-    "properties": {
-        "button_name": "checkout",
-        "page": "product-detail"
-    }
-})
+observe(
+    prompt=user_query,
+    response=message.content[0].text,
+    provider_response=message,  # auto-extracts model, tokens
+)
+```
+
+### RAG applications — grounding & hallucination scoring
 
-# Clean up resources
-client.close()
+```python
+observe(
+    prompt=user_query,
+    response=llm_answer,
+    provider_response=completion,
+    reference_materials=[
+        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
+        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
+    ],
+    retrieval_query=user_query,
+)
 ```
 
+### Multi-turn chat — coherence scoring
+
+```python
+observe(
+    prompt=latest_user_message,
+    response=llm_answer,
+    provider_response=completion,
+    conversation_history=[
+        {"role": "user", "content": "What is diabetes?"},
+        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
+        {"role": "user", "content": "What are the symptoms?"},
+    ],
+    session_id="conv-123",
+)
+```
+
+### RAG applications — grounding & hallucination scoring
+
+```python
+observe(
+    prompt=user_query,
+    response=llm_answer,
+    provider_response=completion,
+    reference_materials=[
+        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
+        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
+    ],
+    retrieval_query=user_query,
+)
+```
+
+### Multi-turn chat — coherence scoring
+
+```python
+observe(
+    prompt=latest_user_message,
+    response=llm_answer,
+    provider_response=completion,
+    conversation_history=[
+        {"role": "user", "content": "What is diabetes?"},
+        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
+        {"role": "user", "content": "What are the symptoms?"},
+    ],
+    session_id="conv-123",
+)
+```
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `prompt` | str | Yes | The user's input / question |
+| `response` | str | Yes | The LLM's generated response |
+| `provider_response` | object | No | Raw OpenAI/Anthropic/Google response — auto-extracts model, tokens |
+| `model` | str | No | Model name (auto-extracted if provider_response given) |
+| `provider` | str | No | "openai", "anthropic", etc. (auto-detected) |
+| `latency_ms` | int | No | Response generation time in milliseconds |
+| `prompt_tokens` | int | No | Input token count (auto-extracted if provider_response given) |
+| `completion_tokens` | int | No | Output token count (auto-extracted) |
+| `cost` | float | No | Cost in USD |
+| `reference_materials` | list[dict] | No | RAG chunks: `[{"id", "content", "source"}]` — enables grounding scoring |
+| `retrieval_query` | str | No | Query sent to retriever — enables retrieval quality scoring |
+| `conversation_history` | list[dict] | No | Prior turns: `[{"role", "content"}]` — enables coherence scoring |
+| `tags` | list[str] | No | Grouping labels, e.g. `["production", "rag"]` |
+| `user_id` | str | No | Your user's ID |
+| `session_id` | str | No | Conversation session ID (groups multi-turn) |
+| `metadata` | dict | No | Any extra key-value data |
+
+---
+
 ## Prompt Experimentation
 
 Variably provides two modes for LLM prompt experimentation:
@@ -248,7 +354,7 @@ result = client.submit_response(
 | `workflow_history[].content` | `str` | Message content |
 | `retrieval_query` | `str` (optional) | The rewritten query sent to the retriever |
 
-See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
+See [Context-Aware RAG Evaluation](../../docs/concepts/LLM%20Evaluation/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
 
 #### Integration with LangGraph / FastAPI streaming
 

{variably_sdk-2.6.0 → variably_sdk-2.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "variably-sdk"
-version = "2.6.0"
+version = "2.7.0"
 description = "Official Python SDK for Variably feature flags, LLM experimentation, and prompt optimization platform"
 readme = "README.md"
 requires-python = ">=3.7"

{variably_sdk-2.6.0 → variably_sdk-2.7.0}/src/variably/client.py

@@ -464,16 +464,26 @@ class VariablyClient:
             experiment_key, dict(user_context), input_variables
         )
 
+        # API returns nested structure: {"experiment": {...}, "selected_variant": {...}}
+        experiment = response.get("experiment", {})
+        variant = response.get("selected_variant", {})
+        params = variant.get("parameters", {})
+
+        # Extract model/provider from nested parameters.llmConfig
+        llm_config = params.get("llmConfig", params)
+        model = llm_config.get("model", "")
+        provider = llm_config.get("provider", "")
+
         return PromptVariant(
-            experiment_id=response["experiment_id"],
-            experiment_key=response["experiment_key"],
-            variant_id=response["variant_id"],
-            variant_key=response["variant_key"],
-            prompt_template=response["prompt_template"],
-            model=response["model"],
-            provider=response["provider"],
-            parameters=response.get("parameters", {}),
-            is_control=response.get("is_control", False),
+            experiment_id=experiment.get("experiment_id", experiment.get("id", "")),
+            experiment_key=experiment_key,
+            variant_id=variant.get("variant_id", variant.get("id", "")),
+            variant_key=variant.get("variant_key", variant.get("name", "")),
+            prompt_template=variant.get("prompt_template", variant.get("prompt", "")),
+            model=model,
+            provider=provider,
+            parameters=params,
+            is_control=variant.get("is_control", variant.get("isControl", False)),
         )
 
     def evaluate_prompt(
@@ -601,41 +611,49 @@ class VariablyClient:
         prompt: Union[str, List[Dict[str, Any]]],
         response: str,
         *,
-        conversation_history: Optional[List[Dict[str, Any]]] = None,
-        reference_materials: Optional[List[Dict[str, Any]]] = None,
-        retrieval_query: Optional[str] = None,
-        tags: Optional[List[str]] = None,
-        user_id: Optional[str] = None,
-        session_id: Optional[str] = None,
-        provider: Optional[str] = None,
-        model: Optional[str] = None,
-        latency_ms: Optional[int] = None,
-        prompt_tokens: Optional[int] = None,
-        completion_tokens: Optional[int] = None,
-        cost: Optional[float] = None,
-        metadata: Optional[Dict[str, Any]] = None,
+        # RAG / Grounding context
+        reference_materials: Optional[List[Dict[str, Any]]] = None,  # Retrieved docs for grounding scoring
+        retrieval_query: Optional[str] = None,     # The actual search query (if rewritten/expanded)
+        # Multi-turn / Session context
+        session_id: Optional[str] = None,          # Groups conversation turns into sessions
+        conversation_history: Optional[List[Dict[str, Any]]] = None,  # Prior turns for coherence scoring
+        # User & organization context
+        user_id: Optional[str] = None,             # End-user ID for per-user analytics
+        tags: Optional[List[str]] = None,          # Grouping labels, e.g. ["production", "rag"]
+        metadata: Optional[Dict[str, Any]] = None, # Arbitrary key-value data
+        # LLM metadata
+        provider: Optional[str] = None,            # e.g. "openai", "anthropic"
+        model: Optional[str] = None,               # e.g. "gpt-4o", "claude-3.5-sonnet"
+        latency_ms: Optional[int] = None,          # Response time in ms
+        prompt_tokens: Optional[int] = None,       # Input token count
+        completion_tokens: Optional[int] = None,   # Output token count
+        cost: Optional[float] = None,              # Cost in USD
     ) -> LogResult:
-        """Log a prompt/response pair for 43-dimension evaluation (Observe Mode).
+        """Log a prompt/response pair for multi-dimensional evaluation (Observe Mode).
 
         Zero-friction entry point — no experiment creation needed. Just log your
         prompts and responses and see quality scores in the dashboard.
 
         Args:
-            prompt: The prompt sent to the LLM (str or messages list).
-            response: The LLM's response text.
-            conversation_history: Prior conversation turns for cross-turn coherence scoring.
-            reference_materials: Retrieved RAG chunks for grounding/hallucination scoring.
-            retrieval_query: The query sent to the retriever (for retrieval relevance scoring).
-            tags: Optional tags for grouping observations (e.g. ["production", "chatbot"]).
-            user_id: End-user ID for per-user analytics.
-            session_id: Session ID for multi-turn tracking.
+            prompt: The user's input / question (str or messages list).
+            response: The AI's generated response text.
+            reference_materials: Retrieved docs (RAG chunks) for grounding scoring.
+                Enables: faithfulness, hallucination rate, attribution accuracy.
+            retrieval_query: The query sent to your retriever/search.
+                If your pipeline rewrites queries, pass the rewritten version.
+            session_id: Conversation/thread ID — groups turns into sessions.
+            conversation_history: Prior turns for coherence scoring.
+                Each dict: {"role": "user"/"assistant", "content": "..."}.
+                Not needed if session_id is provided (auto-built from stored turns).
+            user_id: End-user ID for per-user quality analytics.
+            tags: Grouping labels for filtering (e.g. ["production", "chatbot"]).
+            metadata: Arbitrary key-value data attached to this evaluation.
             provider: LLM provider name (e.g. "openai", "anthropic").
-            model: Model name (e.g. "gpt-4", "claude-3-opus").
+            model: Model name (e.g. "gpt-4o", "claude-3.5-sonnet").
             latency_ms: LLM call latency in milliseconds.
             prompt_tokens: Number of input tokens.
             completion_tokens: Number of output tokens.
             cost: Cost in USD for this LLM call.
-            metadata: Arbitrary key-value metadata.
 
         Returns:
             LogResult with observation_id and status.

{variably_sdk-2.6.0 → variably_sdk-2.7.0}/src/variably/observe.py

@@ -142,58 +142,85 @@ def observe(
     prompt: str,
     response: str,
     *,
-    # Auto-extraction from provider response object
+    # --- Auto-extraction ---
+    # Pass the raw LLM response object (OpenAI, Anthropic, Google).
+    # Model name, token counts, and provider are auto-detected — no manual setup needed.
     provider_response: Any = None,
-    # Manual overrides (take precedence over auto-extracted)
-    model: Optional[str] = None,
-    provider: Optional[str] = None,
-    latency_ms: Optional[int] = None,
-    prompt_tokens: Optional[int] = None,
-    completion_tokens: Optional[int] = None,
-    cost: Optional[float] = None,
-    # RAG context
-    reference_materials: Optional[List[Dict[str, Any]]] = None,
+
+    # --- LLM metadata (manual overrides — take precedence over auto-extracted values) ---
+    model: Optional[str] = None,              # e.g. "gpt-4o", "claude-3.5-sonnet"
+    provider: Optional[str] = None,           # e.g. "openai", "anthropic", "google"
+    latency_ms: Optional[int] = None,         # Response time in milliseconds
+    prompt_tokens: Optional[int] = None,      # Input token count
+    completion_tokens: Optional[int] = None,  # Output token count
+    cost: Optional[float] = None,             # Cost in USD for this LLM call
+
+    # --- RAG / Grounding context ---
+    # Pass your retrieved documents to enable grounding scoring:
+    # faithfulness, hallucination rate, attribution accuracy, context utilization.
+    reference_materials: Optional[List[Dict[str, Any]]] = None,  # [{"id": "chunk-1", "content": "...", "source": "docs.pdf"}]
+    # If your pipeline rewrites/expands queries before search, pass the actual search query here.
+    # Enables retrieval relevance scoring. If not provided, the raw prompt is used.
     retrieval_query: Optional[str] = None,
-    # Multi-turn context
-    conversation_history: Optional[List[Dict[str, str]]] = None,
-    # Metadata
-    tags: Optional[List[str]] = None,
-    user_id: Optional[str] = None,
+
+    # --- Multi-turn / Session context ---
+    # Groups conversation turns into sessions. Enables session-level metrics,
+    # quality trend tracking, and automatic coherence scoring across turns.
+    # Use your conversation/thread ID — any unique string per conversation.
     session_id: Optional[str] = None,
-    metadata: Optional[Dict[str, Any]] = None,
+    # Explicit prior turns for coherence scoring. Each dict: {"role": "user"/"assistant", "content": "..."}.
+    # Advanced: not needed if you pass session_id (variA/Bly auto-builds history from previous turns).
+    prior_turns: Optional[List[Dict[str, str]]] = None,
+
+    # --- User & organization context ---
+    user_id: Optional[str] = None,            # Your end-user's ID — enables per-user analytics
+    tags: Optional[List[str]] = None,         # Grouping labels, e.g. ["production", "rag", "v2"]
+    metadata: Optional[Dict[str, Any]] = None,  # Any arbitrary key-value data for filtering
+
+    # --- Backward compatibility ---
+    conversation_history: Optional[List[Dict[str, str]]] = None,  # Deprecated: use prior_turns instead
 ) -> Optional[Dict[str, Any]]:
-    """Log a prompt/response pair to Variably for multi-dimension evaluation.
+    """Log a prompt/response pair to variA/Bly for multi-dimensional evaluation.
+
+    Add this one call after your LLM generates a response. All parameters are
+    optional except ``prompt`` and ``response``. The more context you provide,
+    the deeper the evaluation.
 
     Never raises. Returns ``{"observation_id": "...", "status": "queued"}``
     on success, or ``None`` on any error (missing API key, network failure, etc.).
 
     Args:
-        prompt: The user's input / question sent to the LLM.
-        response: The LLM's generated response text.
-        provider_response: Raw OpenAI / Anthropic / Google response object.
-            Model name, token counts, and provider are auto-extracted via
-            duck typing — no provider SDK imports needed.
-        model: Model name override (e.g. ``"gpt-4o"``).
-            Takes precedence over auto-extracted value.
-        provider: Provider override (e.g. ``"openai"``).
-            Takes precedence over auto-detected value.
+        prompt: The user's input / question — exactly what the user typed.
+        response: The AI's generated response text.
+
+        provider_response: Raw LLM response object (OpenAI, Anthropic, Google).
+            Auto-extracts model name, token counts, and provider via duck typing.
+
+        model: Model name override (e.g. ``"gpt-4o"``). Overrides auto-extracted value.
+        provider: Provider override (e.g. ``"openai"``). Overrides auto-detected value.
         latency_ms: Response generation time in milliseconds.
-        prompt_tokens: Input token count override.
-            Takes precedence over auto-extracted value.
-        completion_tokens: Output token count override.
-            Takes precedence over auto-extracted value.
+        prompt_tokens: Input token count. Overrides auto-extracted value.
+        completion_tokens: Output token count. Overrides auto-extracted value.
         cost: Cost in USD for this LLM call.
-        reference_materials: RAG chunks for grounding / hallucination scoring.
-            Each dict should have ``"id"``, ``"content"``, and optionally
-            ``"source"``.
-        retrieval_query: The query sent to your retriever (enables retrieval
-            quality scoring).
-        conversation_history: Prior turns for coherence scoring.
-            Each dict should have ``"role"`` and ``"content"``.
-        tags: Grouping labels, e.g. ``["production", "rag"]``.
-        user_id: Your end-user's ID.
-        session_id: Conversation session ID (groups multi-turn observations).
-        metadata: Arbitrary key-value data.
+
+        reference_materials: Retrieved documents (RAG chunks) for grounding scoring.
+            Enables: faithfulness, hallucination rate, attribution accuracy.
+            Each dict should have ``"id"``, ``"content"``, and optionally ``"source"``.
+        retrieval_query: The query your pipeline sent to the retriever/search.
+            If your pipeline rewrites queries before search, pass the rewritten version.
+            Enables accurate retrieval relevance scoring.
+
+        session_id: Conversation/thread ID — groups multi-turn interactions into sessions.
+            Enables: session tracing, quality trends, degradation detection, auto-coherence.
+        prior_turns: Explicit prior conversation turns for coherence scoring.
+            Each dict: ``{"role": "user"/"assistant", "content": "..."}``.
+            Not needed if you pass ``session_id`` (history is auto-built from stored turns).
+
+        user_id: Your end-user's ID for per-user quality analytics.
+        tags: Grouping labels for filtering (e.g. ``["production", "rag"]``).
+        metadata: Arbitrary key-value data attached to this evaluation.
+
+        conversation_history: Deprecated — use ``prior_turns`` instead.
 
     Returns:
         Dict with ``observation_id`` and ``status`` on success, or ``None``.
@@ -217,6 +244,9 @@ def observe(
             else auto.get("completion_tokens")
         )
 
+        # Resolve prior_turns: prefer prior_turns, fall back to deprecated conversation_history
+        resolved_prior_turns = prior_turns or conversation_history
+
         result = client.log(
             prompt=prompt,
             response=response,
@@ -228,7 +258,7 @@ def observe(
             cost=cost,
             reference_materials=reference_materials,
             retrieval_query=retrieval_query,
-            conversation_history=conversation_history,
+            conversation_history=resolved_prior_turns,  # client.log still uses conversation_history internally
             tags=tags,
             user_id=user_id,
             session_id=session_id,

{variably_sdk-2.6.0 → variably_sdk-2.7.0}/src/variably/version.py

@@ -1,3 +1,3 @@
 """Version information for Variably SDK."""
 
-__version__ = "2.6.0"
+__version__ = "2.6.1"

{variably_sdk-2.6.0 → variably_sdk-2.7.0}/src/variably_sdk.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: variably-sdk
-Version: 2.6.0
+Version: 2.7.0
 Summary: Official Python SDK for Variably feature flags, LLM experimentation, and prompt optimization platform
 Author: Variably
 Author-email: Variably <support@variably.com>
@@ -45,7 +45,7 @@ Dynamic: requires-python
 
 # Variably Python SDK
 
-Official Python SDK for Variably — feature flags, LLM experimentation, and prompt optimization.
+Official Python SDK for Variably — LLM evaluation, experimentation, and prompt optimization.
 
 ## Installation
 
@@ -53,52 +53,158 @@ Official Python SDK for Variably — feature flags, LLM experimentation, and pro
 pip install variably-sdk
 ```
 
-## Quick Start
+For Docker/Kubernetes deployments, add to your `requirements.txt`:
+
+```
+variably-sdk>=2.6.1
+```
+## Quick Start — Observe Mode
+
+Add **one line** to your existing LLM app and get **multi-dimension evaluation** across 40+ metrics in 6 categories: Quality, Safety, Semantic, Grounding, Coherence, and Advanced.
+
+No experiment setup. No prompt migration. Just log and see scores.
+
+### 1. Set your environment variables
+
+```bash
+VARIABLY_API_KEY=vbl_your_key_here
+VARIABLY_BASE_URL=https://api.variably.tech
+```
+
+### 2. Add one line after your LLM call
 
 ```python
-from variably import VariablyClient
+from variably import observe
 
-# Initialize the client
-client = VariablyClient({
-    "api_key": "your-api-key",
-    "base_url": "https://api.variably.com",  # optional, defaults to localhost:8080
-    "environment": "production"  # optional
-})
+# Your existing code (unchanged)
+response = your_llm_call(user_query)
 
-# Evaluate a boolean feature flag
-user_context = {
-    "user_id": "user-123",
-    "email": "user@example.com",
-    "country": "US"
-}
+# Add this line:
+observe(prompt=user_query, response=response)
+```
+
+### Auto-extract tokens & model from provider response
 
-is_feature_enabled = client.evaluate_flag_bool(
-    "new-checkout-flow",
-    False,  # default value
-    user_context
+```python
+# OpenAI
+from openai import OpenAI
+from variably import observe
+
+client = OpenAI()
+completion = client.chat.completions.create(
+    model="gpt-4o",
+    messages=[{"role": "user", "content": user_query}],
 )
 
-if is_feature_enabled:
-    # Show new checkout flow
-    pass
+observe(
+    prompt=user_query,
+    response=completion.choices[0].message.content,
+    provider_response=completion,  # auto-extracts model, tokens
+)
+```
 
-# Evaluate a feature gate
-has_access = client.evaluate_gate("premium-features", user_context)
+```python
+# Anthropic
+import anthropic
+from variably import observe
+
+client = anthropic.Anthropic()
+message = client.messages.create(
+    model="claude-sonnet-4-20250514",
+    max_tokens=1024,
+    messages=[{"role": "user", "content": user_query}],
+)
 
-# Track events
-client.track({
-    "name": "button_clicked",
-    "user_id": "user-123",
-    "properties": {
-        "button_name": "checkout",
-        "page": "product-detail"
-    }
-})
+observe(
+    prompt=user_query,
+    response=message.content[0].text,
+    provider_response=message,  # auto-extracts model, tokens
+)
+```
+
+### RAG applications — grounding & hallucination scoring
 
-# Clean up resources
-client.close()
+```python
+observe(
+    prompt=user_query,
+    response=llm_answer,
+    provider_response=completion,
+    reference_materials=[
+        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
+        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
+    ],
+    retrieval_query=user_query,
+)
 ```
 
+### Multi-turn chat — coherence scoring
+
+```python
+observe(
+    prompt=latest_user_message,
+    response=llm_answer,
+    provider_response=completion,
+    conversation_history=[
+        {"role": "user", "content": "What is diabetes?"},
+        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
+        {"role": "user", "content": "What are the symptoms?"},
+    ],
+    session_id="conv-123",
+)
+```
+
+### RAG applications — grounding & hallucination scoring
+
+```python
+observe(
+    prompt=user_query,
+    response=llm_answer,
+    provider_response=completion,
+    reference_materials=[
+        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
+        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
+    ],
+    retrieval_query=user_query,
+)
+```
+
+### Multi-turn chat — coherence scoring
+
+```python
+observe(
+    prompt=latest_user_message,
+    response=llm_answer,
+    provider_response=completion,
+    conversation_history=[
+        {"role": "user", "content": "What is diabetes?"},
+        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
+        {"role": "user", "content": "What are the symptoms?"},
+    ],
+    session_id="conv-123",
+)
+```
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `prompt` | str | Yes | The user's input / question |
+| `response` | str | Yes | The LLM's generated response |
+| `provider_response` | object | No | Raw OpenAI/Anthropic/Google response — auto-extracts model, tokens |
+| `model` | str | No | Model name (auto-extracted if provider_response given) |
+| `provider` | str | No | "openai", "anthropic", etc. (auto-detected) |
+| `latency_ms` | int | No | Response generation time in milliseconds |
+| `prompt_tokens` | int | No | Input token count (auto-extracted if provider_response given) |
+| `completion_tokens` | int | No | Output token count (auto-extracted) |
+| `cost` | float | No | Cost in USD |
+| `reference_materials` | list[dict] | No | RAG chunks: `[{"id", "content", "source"}]` — enables grounding scoring |
+| `retrieval_query` | str | No | Query sent to retriever — enables retrieval quality scoring |
+| `conversation_history` | list[dict] | No | Prior turns: `[{"role", "content"}]` — enables coherence scoring |
+| `tags` | list[str] | No | Grouping labels, e.g. `["production", "rag"]` |
+| `user_id` | str | No | Your user's ID |
+| `session_id` | str | No | Conversation session ID (groups multi-turn) |
+| `metadata` | dict | No | Any extra key-value data |
+
+---
+
 ## Prompt Experimentation
 
 Variably provides two modes for LLM prompt experimentation:
@@ -293,7 +399,7 @@ result = client.submit_response(
 | `workflow_history[].content` | `str` | Message content |
 | `retrieval_query` | `str` (optional) | The rewritten query sent to the retriever |
 
-See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
+See [Context-Aware RAG Evaluation](../../docs/concepts/LLM%20Evaluation/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
 
 #### Integration with LangGraph / FastAPI streaming