variably-sdk 2.1.0__tar.gz → 2.3.0__tar.gz

{variably_sdk-2.1.0 → variably_sdk-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: variably-sdk
-Version: 2.1.0
+Version: 2.3.0
 Summary: Official Python SDK for Variably feature flags, LLM experimentation, and prompt optimization platform
 Author: Variably
 Author-email: Variably <support@variably.com>
@@ -187,6 +187,114 @@ if meta:
     print(f"Tokens: {meta.token_usage}")
 ```
 
+### Context-Aware Evaluation (Better RAG Quality) — v2.2.0+
+
+For RAG chatbots, passing conversation history and retrieved chunks enables **groundedness scoring, hallucination detection, and conversational coherence** — dimensions that are impossible to evaluate in isolation.
+
+The `evaluation_context` parameter is **not sent to the LLM** — it's only used by Variably's evaluator for richer scoring.
+
+```python
+# Step 1: Collect conversation history from your session
+workflow_history = [
+    {"role": "user", "content": "What causes diabetes?"},
+    {"role": "assistant", "content": "Key factors include genetics, diet..."},
+    {"role": "user", "content": "What about potatoes?"},
+]
+
+# Step 2: Collect retrieved RAG chunks (after your retrieval step)
+reference_materials = [
+    {
+        "id": "chunk-001",
+        "content": "Unhealthy diets high in refined sugars, fats...",
+        "source": "Kenya National Clinical Guidelines",
+        "type": "chunk",
+        "relevance_score": 0.89,
+    },
+    {
+        "id": "chunk-002",
+        "content": "Modifiable risk factors include obesity...",
+        "source": "Kenya National Clinical Guidelines",
+        "type": "chunk",
+        "relevance_score": 0.82,
+    },
+]
+
+# Step 3: Pass evaluation_context in your evaluate call
+response = client.evaluate_prompt(
+    experiment_key="rag-prompt-experiment",
+    user_context={"user_id": "user-123"},
+    input_variables={"query": "What about potatoes?", "context": context_text},
+    evaluation_mode="full",
+    evaluation_context={
+        "reference_materials": reference_materials,
+        "workflow_history": workflow_history,
+        "retrieval_query": "potato consumption glycemic index diabetes risk",
+    },
+)
+
+# Same works with streaming
+stream = client.evaluate_prompt_stream(
+    experiment_key="rag-prompt-experiment",
+    user_context={"user_id": "user-123"},
+    input_variables={"query": "What about potatoes?", "context": context_text},
+    evaluation_context={
+        "reference_materials": reference_materials,
+        "workflow_history": workflow_history,
+    },
+)
+for token in stream:
+    print(token, end="", flush=True)
+```
+
+**What this enables:**
+
+| Dimension | Description | Requires |
+|-----------|-------------|----------|
+| `faithfulness` | % of claims grounded in retrieved chunks | `reference_materials` |
+| `hallucination_rate` | % of claims with no source in context | `reference_materials` |
+| `context_utilization` | % of relevant chunks actually used | `reference_materials` |
+| `attribution_accuracy` | Do citations map to correct chunks? | `reference_materials` |
+| `conversation_consistency` | No contradictions with prior turns | `workflow_history` |
+| `context_retention` | Maintains topic awareness across turns | `workflow_history` |
+| `transparency` | Discloses when going beyond source material | `reference_materials` |
+
+**BYOR mode** also supports `evaluation_context` — pass it in `submit_response()`:
+
+```python
+result = client.submit_response(
+    experiment_key="my-experiment",
+    variant_key=variant.variant_key,
+    executed_prompt=prompt,
+    response=llm_response,
+    user_context=user_context,
+    input_variables=input_variables,
+    provider=variant.provider,
+    model=variant.model,
+    latency_ms=latency,
+    evaluation_context={
+        "reference_materials": reference_materials,
+        "workflow_history": workflow_history,
+    },
+)
+```
+
+#### evaluation_context Schema
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `reference_materials` | `list[dict]` | RAG chunks / source documents for groundedness scoring |
+| `reference_materials[].id` | `str` | Unique chunk identifier |
+| `reference_materials[].content` | `str` | Chunk text content |
+| `reference_materials[].source` | `str` (optional) | Source document URL or name |
+| `reference_materials[].type` | `str` (optional) | e.g. `"chunk"`, `"document"` |
+| `reference_materials[].relevance_score` | `float` (optional) | Retriever similarity score |
+| `workflow_history` | `list[dict]` | Conversation turns for coherence scoring |
+| `workflow_history[].role` | `str` | `"user"` or `"assistant"` |
+| `workflow_history[].content` | `str` | Message content |
+| `retrieval_query` | `str` (optional) | The rewritten query sent to the retriever |
+
+See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
+
 #### Integration with LangGraph / FastAPI streaming
 
 ```python
@@ -240,6 +348,11 @@ Content-Type: application/json
   },
   "input_variables": {
     "query": "What are the symptoms of Type 2 diabetes?"
+  },
+  "evaluation_context": {
+    "reference_materials": [{"id": "chunk-1", "content": "...", "source": "...", "type": "chunk"}],
+    "workflow_history": [{"role": "user", "content": "..."}],
+    "retrieval_query": "diabetes symptoms type 2"
   }
 }
 ```

{variably_sdk-2.1.0 → variably_sdk-2.3.0}/README.md

@@ -142,6 +142,114 @@ if meta:
     print(f"Tokens: {meta.token_usage}")
 ```
 
+### Context-Aware Evaluation (Better RAG Quality) — v2.2.0+
+
+For RAG chatbots, passing conversation history and retrieved chunks enables **groundedness scoring, hallucination detection, and conversational coherence** — dimensions that are impossible to evaluate in isolation.
+
+The `evaluation_context` parameter is **not sent to the LLM** — it's only used by Variably's evaluator for richer scoring.
+
+```python
+# Step 1: Collect conversation history from your session
+workflow_history = [
+    {"role": "user", "content": "What causes diabetes?"},
+    {"role": "assistant", "content": "Key factors include genetics, diet..."},
+    {"role": "user", "content": "What about potatoes?"},
+]
+
+# Step 2: Collect retrieved RAG chunks (after your retrieval step)
+reference_materials = [
+    {
+        "id": "chunk-001",
+        "content": "Unhealthy diets high in refined sugars, fats...",
+        "source": "Kenya National Clinical Guidelines",
+        "type": "chunk",
+        "relevance_score": 0.89,
+    },
+    {
+        "id": "chunk-002",
+        "content": "Modifiable risk factors include obesity...",
+        "source": "Kenya National Clinical Guidelines",
+        "type": "chunk",
+        "relevance_score": 0.82,
+    },
+]
+
+# Step 3: Pass evaluation_context in your evaluate call
+response = client.evaluate_prompt(
+    experiment_key="rag-prompt-experiment",
+    user_context={"user_id": "user-123"},
+    input_variables={"query": "What about potatoes?", "context": context_text},
+    evaluation_mode="full",
+    evaluation_context={
+        "reference_materials": reference_materials,
+        "workflow_history": workflow_history,
+        "retrieval_query": "potato consumption glycemic index diabetes risk",
+    },
+)
+
+# Same works with streaming
+stream = client.evaluate_prompt_stream(
+    experiment_key="rag-prompt-experiment",
+    user_context={"user_id": "user-123"},
+    input_variables={"query": "What about potatoes?", "context": context_text},
+    evaluation_context={
+        "reference_materials": reference_materials,
+        "workflow_history": workflow_history,
+    },
+)
+for token in stream:
+    print(token, end="", flush=True)
+```
+
+**What this enables:**
+
+| Dimension | Description | Requires |
+|-----------|-------------|----------|
+| `faithfulness` | % of claims grounded in retrieved chunks | `reference_materials` |
+| `hallucination_rate` | % of claims with no source in context | `reference_materials` |
+| `context_utilization` | % of relevant chunks actually used | `reference_materials` |
+| `attribution_accuracy` | Do citations map to correct chunks? | `reference_materials` |
+| `conversation_consistency` | No contradictions with prior turns | `workflow_history` |
+| `context_retention` | Maintains topic awareness across turns | `workflow_history` |
+| `transparency` | Discloses when going beyond source material | `reference_materials` |
+
+**BYOR mode** also supports `evaluation_context` — pass it in `submit_response()`:
+
+```python
+result = client.submit_response(
+    experiment_key="my-experiment",
+    variant_key=variant.variant_key,
+    executed_prompt=prompt,
+    response=llm_response,
+    user_context=user_context,
+    input_variables=input_variables,
+    provider=variant.provider,
+    model=variant.model,
+    latency_ms=latency,
+    evaluation_context={
+        "reference_materials": reference_materials,
+        "workflow_history": workflow_history,
+    },
+)
+```
+
+#### evaluation_context Schema
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `reference_materials` | `list[dict]` | RAG chunks / source documents for groundedness scoring |
+| `reference_materials[].id` | `str` | Unique chunk identifier |
+| `reference_materials[].content` | `str` | Chunk text content |
+| `reference_materials[].source` | `str` (optional) | Source document URL or name |
+| `reference_materials[].type` | `str` (optional) | e.g. `"chunk"`, `"document"` |
+| `reference_materials[].relevance_score` | `float` (optional) | Retriever similarity score |
+| `workflow_history` | `list[dict]` | Conversation turns for coherence scoring |
+| `workflow_history[].role` | `str` | `"user"` or `"assistant"` |
+| `workflow_history[].content` | `str` | Message content |
+| `retrieval_query` | `str` (optional) | The rewritten query sent to the retriever |
+
+See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
+
 #### Integration with LangGraph / FastAPI streaming
 
 ```python
@@ -195,6 +303,11 @@ Content-Type: application/json
   },
   "input_variables": {
     "query": "What are the symptoms of Type 2 diabetes?"
+  },
+  "evaluation_context": {
+    "reference_materials": [{"id": "chunk-1", "content": "...", "source": "...", "type": "chunk"}],
+    "workflow_history": [{"role": "user", "content": "..."}],
+    "retrieval_query": "diabetes symptoms type 2"
   }
 }
 ```

{variably_sdk-2.1.0 → variably_sdk-2.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "variably-sdk"
-version = "2.1.0"
+version = "2.3.0"
 description = "Official Python SDK for Variably feature flags, LLM experimentation, and prompt optimization platform"
 readme = "README.md"
 requires-python = ">=3.7"

{variably_sdk-2.1.0 → variably_sdk-2.3.0}/src/variably/client.py

@@ -480,7 +480,8 @@ class VariablyClient:
         experiment_key: str,
         user_context: UserContext,
         input_variables: Dict[str, Any],
-        evaluation_mode: str = "full",
+        evaluation_context: Optional[Dict[str, Any]] = None,
+        **kwargs,
     ) -> EvaluateResponse:
         """Managed execution: Variably selects variant, calls LLM, evaluates."""
         if not experiment_key or not isinstance(experiment_key, str):
@@ -493,7 +494,8 @@ class VariablyClient:
             )
 
         response = self.http_client.evaluate_prompt(
-            experiment_key, dict(user_context), input_variables, evaluation_mode
+            experiment_key, dict(user_context), input_variables,
+            evaluation_context=evaluation_context,
         )
 
         return EvaluateResponse(
@@ -513,7 +515,8 @@ class VariablyClient:
         experiment_key: str,
         user_context: UserContext,
         input_variables: Dict[str, Any],
-        evaluation_mode: str = "full",
+        evaluation_context: Optional[Dict[str, Any]] = None,
+        **kwargs,
     ) -> StreamingEvaluateResponse:
         """Managed execution with streaming. Yields token strings.
 
@@ -534,7 +537,8 @@ class VariablyClient:
             )
 
         raw_response = self.http_client.evaluate_prompt_stream(
-            experiment_key, dict(user_context), input_variables, evaluation_mode
+            experiment_key, dict(user_context), input_variables,
+            evaluation_context=evaluation_context,
         )
 
         return StreamingEvaluateResponse(raw_response)
@@ -552,6 +556,7 @@ class VariablyClient:
         latency_ms: Optional[int] = None,
         prompt_tokens: Optional[int] = None,
         completion_tokens: Optional[int] = None,
+        evaluation_context: Optional[Dict[str, Any]] = None,
     ) -> SubmitResponseResult:
         """Submit your own LLM response for 41-dimensional evaluation (BYOR step 3)."""
         if not experiment_key or not isinstance(experiment_key, str):
@@ -583,6 +588,8 @@ class VariablyClient:
             data["prompt_tokens"] = prompt_tokens
         if completion_tokens is not None:
             data["completion_tokens"] = completion_tokens
+        if evaluation_context:
+            data["evaluation_context"] = evaluation_context
 
         result = self.http_client.submit_response(data)
 

{variably_sdk-2.1.0 → variably_sdk-2.3.0}/src/variably/http_client.py

@@ -60,7 +60,7 @@ class HTTPClient:
         session.headers.update({
             "Content-Type": "application/json",
             "X-API-Key": self.config.api_key,
-            "User-Agent": "Variably-Python-SDK/2.1.0",
+            "User-Agent": "Variably-Python-SDK/2.3.0",
         })
         
         return session
@@ -185,15 +185,16 @@ class HTTPClient:
     def evaluate_prompt(
         self, experiment_key: str, user_context: Dict[str, Any],
         input_variables: Dict[str, Any],
-        evaluation_mode: str = "full",
+        evaluation_context: Optional[Dict[str, Any]] = None,
     ) -> Dict[str, Any]:
         """Managed execution: select variant, call LLM, evaluate."""
         request_data = {
             "experiment_key": experiment_key,
             "user_context": self._convert_user_context(user_context),
             "input_variables": input_variables,
-            "evaluation_mode": evaluation_mode,
         }
+        if evaluation_context:
+            request_data["evaluation_context"] = evaluation_context
 
         return self._make_request(
             "POST",
@@ -205,15 +206,16 @@ class HTTPClient:
     def evaluate_prompt_stream(
         self, experiment_key: str, user_context: Dict[str, Any],
         input_variables: Dict[str, Any],
-        evaluation_mode: str = "full",
+        evaluation_context: Optional[Dict[str, Any]] = None,
     ) -> requests.Response:
         """Managed execution with streaming: returns raw SSE response for token parsing."""
         request_data = {
             "experiment_key": experiment_key,
             "user_context": self._convert_user_context(user_context),
             "input_variables": input_variables,
-            "evaluation_mode": evaluation_mode,
         }
+        if evaluation_context:
+            request_data["evaluation_context"] = evaluation_context
 
         url = f"{self.config.base_url}/api/v1/internal/sdk/prompt-experiments/evaluate-stream"
 

{variably_sdk-2.1.0 → variably_sdk-2.3.0}/src/variably/version.py

@@ -1,3 +1,3 @@
 """Version information for Variably SDK."""
 
-__version__ = "2.1.0"
+__version__ = "2.3.0"

{variably_sdk-2.1.0 → variably_sdk-2.3.0}/src/variably_sdk.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: variably-sdk
-Version: 2.1.0
+Version: 2.3.0
 Summary: Official Python SDK for Variably feature flags, LLM experimentation, and prompt optimization platform
 Author: Variably
 Author-email: Variably <support@variably.com>
@@ -187,6 +187,114 @@ if meta:
     print(f"Tokens: {meta.token_usage}")
 ```
 
+### Context-Aware Evaluation (Better RAG Quality) — v2.2.0+
+
+For RAG chatbots, passing conversation history and retrieved chunks enables **groundedness scoring, hallucination detection, and conversational coherence** — dimensions that are impossible to evaluate in isolation.
+
+The `evaluation_context` parameter is **not sent to the LLM** — it's only used by Variably's evaluator for richer scoring.
+
+```python
+# Step 1: Collect conversation history from your session
+workflow_history = [
+    {"role": "user", "content": "What causes diabetes?"},
+    {"role": "assistant", "content": "Key factors include genetics, diet..."},
+    {"role": "user", "content": "What about potatoes?"},
+]
+
+# Step 2: Collect retrieved RAG chunks (after your retrieval step)
+reference_materials = [
+    {
+        "id": "chunk-001",
+        "content": "Unhealthy diets high in refined sugars, fats...",
+        "source": "Kenya National Clinical Guidelines",
+        "type": "chunk",
+        "relevance_score": 0.89,
+    },
+    {
+        "id": "chunk-002",
+        "content": "Modifiable risk factors include obesity...",
+        "source": "Kenya National Clinical Guidelines",
+        "type": "chunk",
+        "relevance_score": 0.82,
+    },
+]
+
+# Step 3: Pass evaluation_context in your evaluate call
+response = client.evaluate_prompt(
+    experiment_key="rag-prompt-experiment",
+    user_context={"user_id": "user-123"},
+    input_variables={"query": "What about potatoes?", "context": context_text},
+    evaluation_mode="full",
+    evaluation_context={
+        "reference_materials": reference_materials,
+        "workflow_history": workflow_history,
+        "retrieval_query": "potato consumption glycemic index diabetes risk",
+    },
+)
+
+# Same works with streaming
+stream = client.evaluate_prompt_stream(
+    experiment_key="rag-prompt-experiment",
+    user_context={"user_id": "user-123"},
+    input_variables={"query": "What about potatoes?", "context": context_text},
+    evaluation_context={
+        "reference_materials": reference_materials,
+        "workflow_history": workflow_history,
+    },
+)
+for token in stream:
+    print(token, end="", flush=True)
+```
+
+**What this enables:**
+
+| Dimension | Description | Requires |
+|-----------|-------------|----------|
+| `faithfulness` | % of claims grounded in retrieved chunks | `reference_materials` |
+| `hallucination_rate` | % of claims with no source in context | `reference_materials` |
+| `context_utilization` | % of relevant chunks actually used | `reference_materials` |
+| `attribution_accuracy` | Do citations map to correct chunks? | `reference_materials` |
+| `conversation_consistency` | No contradictions with prior turns | `workflow_history` |
+| `context_retention` | Maintains topic awareness across turns | `workflow_history` |
+| `transparency` | Discloses when going beyond source material | `reference_materials` |
+
+**BYOR mode** also supports `evaluation_context` — pass it in `submit_response()`:
+
+```python
+result = client.submit_response(
+    experiment_key="my-experiment",
+    variant_key=variant.variant_key,
+    executed_prompt=prompt,
+    response=llm_response,
+    user_context=user_context,
+    input_variables=input_variables,
+    provider=variant.provider,
+    model=variant.model,
+    latency_ms=latency,
+    evaluation_context={
+        "reference_materials": reference_materials,
+        "workflow_history": workflow_history,
+    },
+)
+```
+
+#### evaluation_context Schema
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `reference_materials` | `list[dict]` | RAG chunks / source documents for groundedness scoring |
+| `reference_materials[].id` | `str` | Unique chunk identifier |
+| `reference_materials[].content` | `str` | Chunk text content |
+| `reference_materials[].source` | `str` (optional) | Source document URL or name |
+| `reference_materials[].type` | `str` (optional) | e.g. `"chunk"`, `"document"` |
+| `reference_materials[].relevance_score` | `float` (optional) | Retriever similarity score |
+| `workflow_history` | `list[dict]` | Conversation turns for coherence scoring |
+| `workflow_history[].role` | `str` | `"user"` or `"assistant"` |
+| `workflow_history[].content` | `str` | Message content |
+| `retrieval_query` | `str` (optional) | The rewritten query sent to the retriever |
+
+See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.
+
 #### Integration with LangGraph / FastAPI streaming
 
 ```python
@@ -240,6 +348,11 @@ Content-Type: application/json
   },
   "input_variables": {
     "query": "What are the symptoms of Type 2 diabetes?"
+  },
+  "evaluation_context": {
+    "reference_materials": [{"id": "chunk-1", "content": "...", "source": "...", "type": "chunk"}],
+    "workflow_history": [{"role": "user", "content": "..."}],
+    "retrieval_query": "diabetes symptoms type 2"
   }
 }
 ```