variably-sdk 2.6.1__tar.gz → 2.7.1__tar.gz

{variably_sdk-2.6.1 → variably_sdk-2.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: variably-sdk
-Version: 2.6.1
+Version: 2.7.1
 Summary: Official Python SDK for Variably feature flags, LLM experimentation, and prompt optimization platform
 Author: Variably
 Author-email: Variably <support@variably.com>
@@ -53,6 +53,11 @@ Official Python SDK for Variably — LLM evaluation, experimentation, and prompt
 pip install variably-sdk
 ```
 
+For Docker/Kubernetes deployments, add to your `requirements.txt`:
+
+```
+variably-sdk>=2.6.1
+```
 ## Quick Start — Observe Mode
 
 Add **one line** to your existing LLM app and get **multi-dimension evaluation** across 40+ metrics in 6 categories: Quality, Safety, Semantic, Grounding, Coherence, and Advanced.
@@ -148,7 +153,36 @@ observe(
 )
 ```
 
-### Full parameter reference
+### RAG applications — grounding & hallucination scoring
+
+```python
+observe(
+    prompt=user_query,
+    response=llm_answer,
+    provider_response=completion,
+    reference_materials=[
+        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
+        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
+    ],
+    retrieval_query=user_query,
+)
+```
+
+### Multi-turn chat — coherence scoring
+
+```python
+observe(
+    prompt=latest_user_message,
+    response=llm_answer,
+    provider_response=completion,
+    conversation_history=[
+        {"role": "user", "content": "What is diabetes?"},
+        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
+        {"role": "user", "content": "What are the symptoms?"},
+    ],
+    session_id="conv-123",
+)
+```
 
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
@@ -365,7 +399,7 @@ result = client.submit_response(
 | `workflow_history[].content` | `str` | Message content |
 | `retrieval_query` | `str` (optional) | The rewritten query sent to the retriever |
 
-See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
+See [Context-Aware RAG Evaluation](../../docs/concepts/LLM%20Evaluation/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
 
 #### Integration with LangGraph / FastAPI streaming
 

{variably_sdk-2.6.1 → variably_sdk-2.7.1}/README.md

@@ -8,6 +8,11 @@ Official Python SDK for Variably — LLM evaluation, experimentation, and prompt
 pip install variably-sdk
 ```
 
+For Docker/Kubernetes deployments, add to your `requirements.txt`:
+
+```
+variably-sdk>=2.6.1
+```
 ## Quick Start — Observe Mode
 
 Add **one line** to your existing LLM app and get **multi-dimension evaluation** across 40+ metrics in 6 categories: Quality, Safety, Semantic, Grounding, Coherence, and Advanced.
@@ -103,7 +108,36 @@ observe(
 )
 ```
 
-### Full parameter reference
+### RAG applications — grounding & hallucination scoring
+
+```python
+observe(
+    prompt=user_query,
+    response=llm_answer,
+    provider_response=completion,
+    reference_materials=[
+        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
+        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
+    ],
+    retrieval_query=user_query,
+)
+```
+
+### Multi-turn chat — coherence scoring
+
+```python
+observe(
+    prompt=latest_user_message,
+    response=llm_answer,
+    provider_response=completion,
+    conversation_history=[
+        {"role": "user", "content": "What is diabetes?"},
+        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
+        {"role": "user", "content": "What are the symptoms?"},
+    ],
+    session_id="conv-123",
+)
+```
 
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
@@ -320,7 +354,7 @@ result = client.submit_response(
 | `workflow_history[].content` | `str` | Message content |
 | `retrieval_query` | `str` (optional) | The rewritten query sent to the retriever |
 
-See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
+See [Context-Aware RAG Evaluation](../../docs/concepts/LLM%20Evaluation/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
 
 #### Integration with LangGraph / FastAPI streaming
 

{variably_sdk-2.6.1 → variably_sdk-2.7.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "variably-sdk"
-version = "2.6.1"
+version = "2.7.1"
 description = "Official Python SDK for Variably feature flags, LLM experimentation, and prompt optimization platform"
 readme = "README.md"
 requires-python = ">=3.7"

{variably_sdk-2.6.1 → variably_sdk-2.7.1}/src/variably/client.py

@@ -464,16 +464,26 @@ class VariablyClient:
             experiment_key, dict(user_context), input_variables
         )
 
+        # API returns nested structure: {"experiment": {...}, "selected_variant": {...}}
+        experiment = response.get("experiment", {})
+        variant = response.get("selected_variant", {})
+        params = variant.get("parameters", {})
+
+        # Extract model/provider from nested parameters.llmConfig
+        llm_config = params.get("llmConfig", params)
+        model = llm_config.get("model", "")
+        provider = llm_config.get("provider", "")
+
         return PromptVariant(
-            experiment_id=response["experiment_id"],
-            experiment_key=response["experiment_key"],
-            variant_id=response["variant_id"],
-            variant_key=response["variant_key"],
-            prompt_template=response["prompt_template"],
-            model=response["model"],
-            provider=response["provider"],
-            parameters=response.get("parameters", {}),
-            is_control=response.get("is_control", False),
+            experiment_id=experiment.get("experiment_id", experiment.get("id", "")),
+            experiment_key=experiment_key,
+            variant_id=variant.get("variant_id", variant.get("id", "")),
+            variant_key=variant.get("variant_key", variant.get("name", "")),
+            prompt_template=variant.get("prompt_template", variant.get("prompt", "")),
+            model=model,
+            provider=provider,
+            parameters=params,
+            is_control=variant.get("is_control", variant.get("isControl", False)),
         )
 
     def evaluate_prompt(
@@ -576,7 +586,8 @@ class VariablyClient:
             "response": response,
         }
         if user_context:
-            data["user_context"] = dict(user_context)
+            # Convert snake_case keys to camelCase for the Go backend
+            data["user_context"] = self.http_client._convert_user_context(dict(user_context))
         if input_variables:
             data["input_variables"] = input_variables
         if provider:
@@ -601,41 +612,49 @@ class VariablyClient:
         prompt: Union[str, List[Dict[str, Any]]],
         response: str,
         *,
-        conversation_history: Optional[List[Dict[str, Any]]] = None,
-        reference_materials: Optional[List[Dict[str, Any]]] = None,
-        retrieval_query: Optional[str] = None,
-        tags: Optional[List[str]] = None,
-        user_id: Optional[str] = None,
-        session_id: Optional[str] = None,
-        provider: Optional[str] = None,
-        model: Optional[str] = None,
-        latency_ms: Optional[int] = None,
-        prompt_tokens: Optional[int] = None,
-        completion_tokens: Optional[int] = None,
-        cost: Optional[float] = None,
-        metadata: Optional[Dict[str, Any]] = None,
+        # RAG / Grounding context
+        reference_materials: Optional[List[Dict[str, Any]]] = None,  # Retrieved docs for grounding scoring
+        retrieval_query: Optional[str] = None,     # The actual search query (if rewritten/expanded)
+        # Multi-turn / Session context
+        session_id: Optional[str] = None,          # Groups conversation turns into sessions
+        conversation_history: Optional[List[Dict[str, Any]]] = None,  # Prior turns for coherence scoring
+        # User & organization context
+        user_id: Optional[str] = None,             # End-user ID for per-user analytics
+        tags: Optional[List[str]] = None,          # Grouping labels, e.g. ["production", "rag"]
+        metadata: Optional[Dict[str, Any]] = None, # Arbitrary key-value data
+        # LLM metadata
+        provider: Optional[str] = None,            # e.g. "openai", "anthropic"
+        model: Optional[str] = None,               # e.g. "gpt-4o", "claude-3.5-sonnet"
+        latency_ms: Optional[int] = None,          # Response time in ms
+        prompt_tokens: Optional[int] = None,       # Input token count
+        completion_tokens: Optional[int] = None,   # Output token count
+        cost: Optional[float] = None,              # Cost in USD
     ) -> LogResult:
-        """Log a prompt/response pair for 43-dimension evaluation (Observe Mode).
+        """Log a prompt/response pair for multi-dimensional evaluation (Observe Mode).
 
         Zero-friction entry point — no experiment creation needed. Just log your
         prompts and responses and see quality scores in the dashboard.
 
         Args:
-            prompt: The prompt sent to the LLM (str or messages list).
-            response: The LLM's response text.
-            conversation_history: Prior conversation turns for cross-turn coherence scoring.
-            reference_materials: Retrieved RAG chunks for grounding/hallucination scoring.
-            retrieval_query: The query sent to the retriever (for retrieval relevance scoring).
-            tags: Optional tags for grouping observations (e.g. ["production", "chatbot"]).
-            user_id: End-user ID for per-user analytics.
-            session_id: Session ID for multi-turn tracking.
+            prompt: The user's input / question (str or messages list).
+            response: The AI's generated response text.
+            reference_materials: Retrieved docs (RAG chunks) for grounding scoring.
+                Enables: faithfulness, hallucination rate, attribution accuracy.
+            retrieval_query: The query sent to your retriever/search.
+                If your pipeline rewrites queries, pass the rewritten version.
+            session_id: Conversation/thread ID — groups turns into sessions.
+            conversation_history: Prior turns for coherence scoring.
+                Each dict: {"role": "user"/"assistant", "content": "..."}.
+                Not needed if session_id is provided (auto-built from stored turns).
+            user_id: End-user ID for per-user quality analytics.
+            tags: Grouping labels for filtering (e.g. ["production", "chatbot"]).
+            metadata: Arbitrary key-value data attached to this evaluation.
             provider: LLM provider name (e.g. "openai", "anthropic").
-            model: Model name (e.g. "gpt-4", "claude-3-opus").
+            model: Model name (e.g. "gpt-4o", "claude-3.5-sonnet").
             latency_ms: LLM call latency in milliseconds.
             prompt_tokens: Number of input tokens.
             completion_tokens: Number of output tokens.
             cost: Cost in USD for this LLM call.
-            metadata: Arbitrary key-value metadata.
 
         Returns:
             LogResult with observation_id and status.

{variably_sdk-2.6.1 → variably_sdk-2.7.1}/src/variably/observe.py

@@ -142,58 +142,85 @@ def observe(
     prompt: str,
     response: str,
     *,
-    # Auto-extraction from provider response object
+    # --- Auto-extraction ---
+    # Pass the raw LLM response object (OpenAI, Anthropic, Google).
+    # Model name, token counts, and provider are auto-detected — no manual setup needed.
     provider_response: Any = None,
-    # Manual overrides (take precedence over auto-extracted)
-    model: Optional[str] = None,
-    provider: Optional[str] = None,
-    latency_ms: Optional[int] = None,
-    prompt_tokens: Optional[int] = None,
-    completion_tokens: Optional[int] = None,
-    cost: Optional[float] = None,
-    # RAG context
-    reference_materials: Optional[List[Dict[str, Any]]] = None,
+
+    # --- LLM metadata (manual overrides — take precedence over auto-extracted values) ---
+    model: Optional[str] = None,              # e.g. "gpt-4o", "claude-3.5-sonnet"
+    provider: Optional[str] = None,           # e.g. "openai", "anthropic", "google"
+    latency_ms: Optional[int] = None,         # Response time in milliseconds
+    prompt_tokens: Optional[int] = None,      # Input token count
+    completion_tokens: Optional[int] = None,  # Output token count
+    cost: Optional[float] = None,             # Cost in USD for this LLM call
+
+    # --- RAG / Grounding context ---
+    # Pass your retrieved documents to enable grounding scoring:
+    # faithfulness, hallucination rate, attribution accuracy, context utilization.
+    reference_materials: Optional[List[Dict[str, Any]]] = None,  # [{"id": "chunk-1", "content": "...", "source": "docs.pdf"}]
+    # If your pipeline rewrites/expands queries before search, pass the actual search query here.
+    # Enables retrieval relevance scoring. If not provided, the raw prompt is used.
     retrieval_query: Optional[str] = None,
-    # Multi-turn context
-    conversation_history: Optional[List[Dict[str, str]]] = None,
-    # Metadata
-    tags: Optional[List[str]] = None,
-    user_id: Optional[str] = None,
+
+    # --- Multi-turn / Session context ---
+    # Groups conversation turns into sessions. Enables session-level metrics,
+    # quality trend tracking, and automatic coherence scoring across turns.
+    # Use your conversation/thread ID — any unique string per conversation.
     session_id: Optional[str] = None,
-    metadata: Optional[Dict[str, Any]] = None,
+    # Explicit prior turns for coherence scoring. Each dict: {"role": "user"/"assistant", "content": "..."}.
+    # Advanced: not needed if you pass session_id (variA/Bly auto-builds history from previous turns).
+    prior_turns: Optional[List[Dict[str, str]]] = None,
+
+    # --- User & organization context ---
+    user_id: Optional[str] = None,            # Your end-user's ID — enables per-user analytics
+    tags: Optional[List[str]] = None,         # Grouping labels, e.g. ["production", "rag", "v2"]
+    metadata: Optional[Dict[str, Any]] = None,  # Any arbitrary key-value data for filtering
+
+    # --- Backward compatibility ---
+    conversation_history: Optional[List[Dict[str, str]]] = None,  # Deprecated: use prior_turns instead
 ) -> Optional[Dict[str, Any]]:
-    """Log a prompt/response pair to Variably for multi-dimension evaluation.
+    """Log a prompt/response pair to variA/Bly for multi-dimensional evaluation.
+
+    Add this one call after your LLM generates a response. All parameters are
+    optional except ``prompt`` and ``response``. The more context you provide,
+    the deeper the evaluation.
 
     Never raises. Returns ``{"observation_id": "...", "status": "queued"}``
     on success, or ``None`` on any error (missing API key, network failure, etc.).
 
     Args:
-        prompt: The user's input / question sent to the LLM.
-        response: The LLM's generated response text.
-        provider_response: Raw OpenAI / Anthropic / Google response object.
-            Model name, token counts, and provider are auto-extracted via
-            duck typing — no provider SDK imports needed.
-        model: Model name override (e.g. ``"gpt-4o"``).
-            Takes precedence over auto-extracted value.
-        provider: Provider override (e.g. ``"openai"``).
-            Takes precedence over auto-detected value.
+        prompt: The user's input / question — exactly what the user typed.
+        response: The AI's generated response text.
+
+        provider_response: Raw LLM response object (OpenAI, Anthropic, Google).
+            Auto-extracts model name, token counts, and provider via duck typing.
+
+        model: Model name override (e.g. ``"gpt-4o"``). Overrides auto-extracted value.
+        provider: Provider override (e.g. ``"openai"``). Overrides auto-detected value.
         latency_ms: Response generation time in milliseconds.
-        prompt_tokens: Input token count override.
-            Takes precedence over auto-extracted value.
-        completion_tokens: Output token count override.
-            Takes precedence over auto-extracted value.
+        prompt_tokens: Input token count. Overrides auto-extracted value.
+        completion_tokens: Output token count. Overrides auto-extracted value.
         cost: Cost in USD for this LLM call.
-        reference_materials: RAG chunks for grounding / hallucination scoring.
-            Each dict should have ``"id"``, ``"content"``, and optionally
-            ``"source"``.
-        retrieval_query: The query sent to your retriever (enables retrieval
-            quality scoring).
-        conversation_history: Prior turns for coherence scoring.
-            Each dict should have ``"role"`` and ``"content"``.
-        tags: Grouping labels, e.g. ``["production", "rag"]``.
-        user_id: Your end-user's ID.
-        session_id: Conversation session ID (groups multi-turn observations).
-        metadata: Arbitrary key-value data.
+
+        reference_materials: Retrieved documents (RAG chunks) for grounding scoring.
+            Enables: faithfulness, hallucination rate, attribution accuracy.
+            Each dict should have ``"id"``, ``"content"``, and optionally ``"source"``.
+        retrieval_query: The query your pipeline sent to the retriever/search.
+            If your pipeline rewrites queries before search, pass the rewritten version.
+            Enables accurate retrieval relevance scoring.
+
+        session_id: Conversation/thread ID — groups multi-turn interactions into sessions.
+            Enables: session tracing, quality trends, degradation detection, auto-coherence.
+        prior_turns: Explicit prior conversation turns for coherence scoring.
+            Each dict: ``{"role": "user"/"assistant", "content": "..."}``.
+            Not needed if you pass ``session_id`` (history is auto-built from stored turns).
+
+        user_id: Your end-user's ID for per-user quality analytics.
+        tags: Grouping labels for filtering (e.g. ``["production", "rag"]``).
+        metadata: Arbitrary key-value data attached to this evaluation.
+
+        conversation_history: Deprecated — use ``prior_turns`` instead.
 
     Returns:
         Dict with ``observation_id`` and ``status`` on success, or ``None``.
@@ -217,6 +244,9 @@ def observe(
             else auto.get("completion_tokens")
         )
 
+        # Resolve prior_turns: prefer prior_turns, fall back to deprecated conversation_history
+        resolved_prior_turns = prior_turns or conversation_history
+
         result = client.log(
             prompt=prompt,
             response=response,
@@ -228,7 +258,7 @@ def observe(
             cost=cost,
             reference_materials=reference_materials,
             retrieval_query=retrieval_query,
-            conversation_history=conversation_history,
+            conversation_history=resolved_prior_turns,  # client.log still uses conversation_history internally
             tags=tags,
             user_id=user_id,
             session_id=session_id,

{variably_sdk-2.6.1 → variably_sdk-2.7.1}/src/variably_sdk.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: variably-sdk
-Version: 2.6.1
+Version: 2.7.1
 Summary: Official Python SDK for Variably feature flags, LLM experimentation, and prompt optimization platform
 Author: Variably
 Author-email: Variably <support@variably.com>
@@ -53,6 +53,11 @@ Official Python SDK for Variably — LLM evaluation, experimentation, and prompt
 pip install variably-sdk
 ```
 
+For Docker/Kubernetes deployments, add to your `requirements.txt`:
+
+```
+variably-sdk>=2.6.1
+```
 ## Quick Start — Observe Mode
 
 Add **one line** to your existing LLM app and get **multi-dimension evaluation** across 40+ metrics in 6 categories: Quality, Safety, Semantic, Grounding, Coherence, and Advanced.
@@ -148,7 +153,36 @@ observe(
 )
 ```
 
-### Full parameter reference
+### RAG applications — grounding & hallucination scoring
+
+```python
+observe(
+    prompt=user_query,
+    response=llm_answer,
+    provider_response=completion,
+    reference_materials=[
+        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
+        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
+    ],
+    retrieval_query=user_query,
+)
+```
+
+### Multi-turn chat — coherence scoring
+
+```python
+observe(
+    prompt=latest_user_message,
+    response=llm_answer,
+    provider_response=completion,
+    conversation_history=[
+        {"role": "user", "content": "What is diabetes?"},
+        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
+        {"role": "user", "content": "What are the symptoms?"},
+    ],
+    session_id="conv-123",
+)
+```
 
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
@@ -365,7 +399,7 @@ result = client.submit_response(
 | `workflow_history[].content` | `str` | Message content |
 | `retrieval_query` | `str` (optional) | The rewritten query sent to the retriever |
 
-See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
+See [Context-Aware RAG Evaluation](../../docs/concepts/LLM%20Evaluation/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
 
 #### Integration with LangGraph / FastAPI streaming