judgeval 0.0.12__tar.gz → 0.0.13__tar.gz

{judgeval-0.0.12 → judgeval-0.0.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: judgeval
-Version: 0.0.12
+Version: 0.0.13
 Summary: Judgeval Package
 Project-URL: Homepage, https://github.com/JudgmentLabs/judgeval
 Project-URL: Issues, https://github.com/JudgmentLabs/judgeval/issues

{judgeval-0.0.12 → judgeval-0.0.13}/docs/api_reference/trace.mdx

@@ -12,27 +12,34 @@ The `Tracer` class is used to trace the execution of your LLM system.
 ```python
 from judgeval.common.tracer import Tracer
 
-tracer = Tracer()
+tracer = Tracer(project_name="my_project")
 ```
 
 <Note>
-The `Tracer` class is a singleton, so you only need to initialize it once in your application.
+The `Tracer` class is a singleton, so you only need to initialize it once in your application. 
+The `project_name` enables you to group traces by workflow, keeping all your evaluations and 
+observability tooling in one place.
 </Note>
 
-## Exporting traces
+## Explicitly exporting traces
 
 When using the `.trace()` context manager, you can control how your traces are exported to the Judgment platform by 
 providing the `project_name` argument. This allows you to group traces by workflow, keeping all your evaluations and 
 observability tooling in one place.
 
 ```python
-with tracer.trace("my_workflow", project_name="my_project"): 
+with tracer.trace(
+    name="my_workflow", 
+    project_name="my_project", 
+    overwrite=True
+    ) as trace: 
     ...
 ```
 
 `.trace()` has the following args:
 - `name`: The name of the trace. Can be make unique to each workflow run by using a timestamp or other unique identifier.
 - `project_name`: The name of the project to use for the trace. Used to group traces by workflow.
+- `overwrite`: Whether to overwrite the trace with the same `name` if it already exists.
 
 The `trace()` context manager yields a `TraceClient` object.
 

{judgeval-0.0.12 → judgeval-0.0.13}/docs/getting_started.mdx

@@ -32,7 +32,7 @@ large-scale evaluations. [Contact us](mailto:contact@judgmentlabs.ai) if you're
 sensitive data that has to reside in your private VPCs.
 </Note>
 
-# Create your first evaluation
+# Create Your First Evaluation
 
 ```python sample_eval.py
 from judgeval import JudgmentClient
@@ -68,6 +68,48 @@ is a scorer that checks if the output is hallucinated relative to the retrieved
 To learn more about using the Judgment Client to run evaluations, click [here](/api_reference/judgment_client).
 </Tip>
 
+# Create Your First Trace
+
+Beyond experimentation, `judgeval` supports monitoring your LLM systems in **production**. 
+Using our `tracing` module, you can **track your LLM system outputs from end to end**, allowing you to visualize the flow of your LLM system. 
+Additionally, you can **enable evaluations to run in real-time** using Judgment's state-of-the-art judge models. 
+
+```python trace_example.py
+from judgeval.common.tracer import Tracer, wrap
+from openai import OpenAI
+
+client = wrap(OpenAI())
+judgment = Tracer(project_name="my_project")
+
+@judgment.observe(span_type="tool")
+def my_tool():
+    return "Hello world!"
+
+@judgment.observe(span_type="function")
+def main():
+    res = client.chat.completions.create(
+        model="gpt-4o",
+        messages=[{"role": "user", "content": f"{my_tool()}"}]
+    )
+    return res.choices[0].message.content
+```
+
+
+<div style={{display: 'flex', justifyContent: 'center'}}>
+  ![Alt text](/images/trace_screenshot.png "Image of a RAG pipeline trace")
+</div>
+
+There are many benefits of monitoring your LLM systems in production with `judgeval`, including:
+- Detecting hallucinations and other quality issues **before they reach your customers**
+- Automatically creating experimental datasets from your **real-world production cases** for future improvement/optimization
+- Track and create alerts on **any metric** (e.g. latency, cost, hallucination, etc.)
+
+<Tip>
+To learn more about `judgeval`'s tracing module, click [here](/tracing/introduction).
+</Tip>
+
+
+
 # Create Your First Scorer
 `judgeval` offers three kinds of LLM scorers for your evaluation needs: ready-made, classifier scorers, and custom scorers.
 
@@ -264,22 +306,3 @@ A `Project` keeps track of `Evaluation Run`s in your project. Each `Evaluation R
 You can try different models (e.g. `gpt-4o`, `claude-3-5-sonnet`, etc.) and prompt templates in each `Evaluation Run` to find the 
 optimal setup for your LLM system. 
 </Tip>
-
-## Monitoring LLM Systems in Production
-
-Beyond experimenting and measuring historical performance, `judgeval` supports monitoring your LLM systems in **production**. 
-Using our `tracing` module, you can **track your LLM system outputs from end to end**, allowing you to visualize the flow of your LLM system. 
-Additionally, you can **enable evaluations to run in real-time** using Judgment's state-of-the-art judge models. 
-
-<div style={{display: 'flex', justifyContent: 'center'}}>
-  ![Alt text](/images/trace_screenshot.png "Image of a RAG pipeline trace")
-</div>
-
-There are many benefits of monitoring your LLM systems in production with `judgeval`, including:
-- Detecting hallucinations and other quality issues **before they reach your customers**
-- Automatically creating experimental datasets from your **real-world production cases** for future improvement/optimization
-- Track and create alerts on **any metric** (e.g. latency, cost, hallucination, etc.)
-
-<Tip>
-To learn more about `judgeval`'s tracing module, click [here](/tracing/introduction).
-</Tip>

{judgeval-0.0.12 → judgeval-0.0.13}/docs/monitoring/tracing.mdx

@@ -18,24 +18,25 @@ Using tracing, you can:
 
 ## Tracing Your Workflow ##
 
-Setting up tracing with `judgeval` takes three simple steps:
+Setting up tracing with `judgeval` takes two simple steps:
 
-### 1. Initialize a tracer with your API key
+### 1. Initialize a tracer with your API key and project name
 
 ```python
 from judgeval.common.tracer import Tracer
 
-judgment = Tracer()  # loads from JUDGMENT_API_KEY env var
+judgment = Tracer(project_name="my_project")  # loads from JUDGMENT_API_KEY env var
 ```
 
 <Note>
-    The [Judgment tracer](/api_reference/trace) is a singleton object that should be shared across your application.
+    The [Judgment tracer](/api_reference/trace) is a singleton object that should be shared across your application. 
+    Your project name will be used to organize your traces in one place on the Judgment platform.
 </Note>
 
 
 ### 2. Wrap your workflow components
 
-`judgeval` provides three wrapping mechanisms for your workflow components:
+`judgeval` provides wrapping mechanisms for your workflow components:
 
 #### `wrap()` ####
 The `wrap()` function goes over your LLM client (e.g. OpenAI, Anthropic, etc.) and captures metadata surrounding your LLM calls, such as:
@@ -44,6 +45,14 @@ The `wrap()` function goes over your LLM client (e.g. OpenAI, Anthropic, etc.) a
 - Prompt/Completion
 - Model name
 
+Here's an example of using `wrap()` on an OpenAI client:
+```python
+from openai import OpenAI
+from judgeval.common.tracer import wrap
+
+client = wrap(OpenAI())
+```
+
 #### `@observe` ####
 The `@observe` decorator wraps your functions/tools and captures metadata surrounding your function calls, such as:
 - Latency
@@ -63,30 +72,20 @@ def my_tool():
 ```
 
 <Note>
-    The `@observe` decorator is used on top of helper functions that you write, but is not designed to be used 
-    on your "main" function. For more information, see the `context manager` section below.
+    `span_type` is a string that you can use to categorize and organize your trace spans. 
+    Span types are displayed on the trace UI to easily nagivate a visualization of your workflow. 
+    Common span types include `tool`, `function`, `retriever`, `database`, `web search`, etc.
 </Note>
 
-#### `context manager` ####
-
-In your main function (e.g. the one that executes the primary workflow logic), you can use the `with judgment.trace()` context manager to trace the entire workflow.
-
-The context manager can **save/print the state of the trace at any point in the workflow**.
-This is useful for debugging or exporting any state of your workflow to run an evaluation from!
-
-<Tip>
-    The `with judgment.trace()` context manager detects any `@observe` decorated functions or wrapped LLM calls within the context and automatically captures their metadata.
-</Tip>
-
 
 #### Putting it all Together
-Here's a complete example of using the `with judgment.trace()` context manager with the other tracing mechanisms:
+Here's a complete example of using judgeval's tracing mechanisms:
 ```python
 from judgeval.common.tracer import Tracer, wrap
 from openai import OpenAI
 
 openai_client = wrap(OpenAI())
-judgment = Tracer()  # loads from JUDGMENT_API_KEY env var
+judgment = Tracer(project_name="my_project")  # loads from JUDGMENT_API_KEY env var
 
 @judgment.observe(span_type="tool")
 def my_tool():
@@ -101,28 +100,10 @@ def my_llm_call():
     )
     return res.choices[0].message.content
 
+@judgment.observe(span_type="function")
 def main():
-    with judgment.trace(
-        "main_workflow", 
-        project_name="my_project"
-    ) as trace:
-        res = my_llm_call()
-        trace.save()
-        trace.print()
-        return res
-```
-
-The printed trace appears as follows on the terminal:
-```
-→ main_workflow (trace: main_workflow)
-  → my_llm_call (trace: my_llm_call)
-    Input: {'args': [], 'kwargs': {}}
-    → my_tool (trace: my_tool)
-      Input: {'args': [], 'kwargs': {}}
-      Output: Hello world!
-    ← my_tool (0.000s)
-    Output: Hello! How can I assist you today?
-  ← my_llm_call (0.789s)
+    res = my_llm_call()
+    return res
 ```
 
 And the trace will appear on the Judgment platform as follows:
@@ -142,32 +123,27 @@ To execute an asynchronous evaluation, you can use the `trace.async_evaluate()`
 
 ```python
 from judgeval.common.tracer import Tracer
-from judgeval.scorers import FaithfulnessScorer
+from judgeval.scorers import AnswerRelevancyScorer
 
-judgment = Tracer()
+judgment = Tracer(project_name="my_project")
 
+@judgment.observe(span_type="function")
 def main():
-    with judgment.trace(
-        "main_workflow", 
-        project_name="my_project"
-    ) as trace:
-        retrieved_info = ...   # from knowledge base
-        res = ...  # your main workflow logic
-        
-        judgment.get_current_trace().async_evaluate(
-            scorers=[FaithfulnesssScorer(threshold=0.5)],
-            input="",
-            actual_output=res,
-            retrieval_context=[retrieved_info],
-            model="gpt-4o-mini",
-        )
-        return res
+    query = "What is the capital of France?"
+    res = ...  # Your workflow logic
+    
+    judgment.get_current_trace().async_evaluate(
+        scorers=[AnswerRelevancyScorer(threshold=0.5)],
+        input="",
+        actual_output=res,
+        model="gpt-4o",
+    )
+    return res
 ```
 
 <Tip>
-You can organize how your async evaluation runs are logged to the Judgment platform by using the 
-`project_name` argument in the `trace` context manager. See our [API documentation](/api_reference/trace) 
-for more information.
+    Your async evaluations will be logged to the Judgment platform as part of the original trace and 
+    a new evaluation will be created on the Judgment platform.
 </Tip>
 
 ## Example: OpenAI Travel Agent
@@ -183,4 +159,53 @@ In this video, we'll walk through all of the topics covered in this guide by tra
     allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" 
     referrerpolicy="strict-origin-when-cross-origin" 
     allowfullscreen
-></iframe>
+></iframe>
+
+
+## Advanced: Customizing Traces Using the Context Manager ##
+
+If you need to customize your tracing context, you can use the `with judgment.trace()` context manager.
+
+The context manager can **save/print the state of the trace at any point in the workflow**.
+This is useful for debugging or exporting any state of your workflow to run an evaluation from!
+
+<Tip>
+    The `with judgment.trace()` context manager detects any `@observe` decorated functions or wrapped LLM calls within the context and automatically captures their metadata.
+</Tip>
+
+Here's an example of using the context manager to trace a workflow:
+```python
+from judgeval.common.tracer import Tracer, wrap
+from openai import OpenAI
+
+judgment = Tracer(project_name="my_project")
+client = wrap(OpenAI())
+
+@judgment.observe(span_type="tool")
+def my_tool():
+    return "Hello world!"
+
+def main():
+    with judgment.trace(name="my_workflow") as trace:
+        res = client.chat.completions.create(
+            model="gpt-4o",
+            messages=[{"role": "user", "content": f"{my_tool()}"}]
+        )
+    
+    trace.print()  # prints the state of the trace to console
+    trace.save()  # saves the current state of the trace to the Judgment platform
+
+    return res.choices[0].message.content
+```
+
+<Warning>
+    The `with judgment.trace()` context manager should only be used if you need to customize the context 
+    over which you're tracing. In most cases, you should trace using the `@observe` decorator.
+</Warning>
+
+
+
+
+
+
+

{judgeval-0.0.12 → judgeval-0.0.13}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "judgeval"
-version = "0.0.12"
+version = "0.0.13"
 authors = [
     { name="Andrew Li", email="andrew@judgmentlabs.ai" },
     { name="Alex Shan", email="alex@judgmentlabs.ai" },

{judgeval-0.0.12 → judgeval-0.0.13}/src/judgeval/common/tracer.py

@@ -199,10 +199,11 @@ class TraceManagerClient:
             JUDGMENT_TRACES_FETCH_API_URL,
             json={
                 "trace_id": trace_id,
-                "judgment_api_key": self.judgment_api_key,
+                # "judgment_api_key": self.judgment_api_key,
             },
             headers={
                 "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.judgment_api_key}"
             }
         )
 
@@ -225,6 +226,7 @@ class TraceManagerClient:
             json=trace_data,
             headers={
                 "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.judgment_api_key}"
             }
         )
         
@@ -248,6 +250,7 @@ class TraceManagerClient:
             },
             headers={
                 "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.judgment_api_key}"
             }
         )
 
@@ -263,11 +266,12 @@ class TraceManagerClient:
         response = requests.delete(
             JUDGMENT_TRACES_DELETE_API_URL,
             json={
-                "judgment_api_key": self.judgment_api_key,
+                # "judgment_api_key": self.judgment_api_key,
                 "trace_ids": trace_ids,
             },
             headers={
                 "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.judgment_api_key}"
             }
         )
 
@@ -576,6 +580,25 @@ class TraceClient:
         
         self.trace_manager_client.save_trace(trace_data, empty_save)
 
+
+        # Save trace data by making POST request to API
+        response = requests.post(
+            JUDGMENT_TRACES_SAVE_API_URL,
+            json=trace_data,
+            headers={
+                "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.tracer.api_key}"  # Bearer token format
+            }
+        )
+        
+        if response.status_code == HTTPStatus.BAD_REQUEST:
+            raise ValueError(f"Failed to save trace data: Check your Trace name for conflicts, set overwrite=True to overwrite existing traces: {response.text}")
+        elif response.status_code != HTTPStatus.OK:
+            raise ValueError(f"Failed to save trace data: {response.text}")
+        
+        if not empty_save and "ui_results_url" in response.json():
+            rprint(f"\n🔍 You can view your trace data here: [rgb(106,0,255)]{response.json()['ui_results_url']}[/]\n")
+        
         return self.trace_id, trace_data
 
     def delete(self):

{judgeval-0.0.12 → judgeval-0.0.13}/src/judgeval/constants.py

@@ -36,7 +36,9 @@ ROOT_API = os.getenv("JUDGMENT_API_URL", "https://api.judgmentlabs.ai")
 JUDGMENT_EVAL_API_URL = f"{ROOT_API}/evaluate/"
 JUDGMENT_DATASETS_PUSH_API_URL = f"{ROOT_API}/datasets/push/"
 JUDGMENT_DATASETS_PULL_API_URL = f"{ROOT_API}/datasets/pull/"
+JUDGMENT_DATASETS_EXPORT_JSONL_API_URL = f"{ROOT_API}/datasets/export_jsonl/"
 JUDGMENT_DATASETS_PULL_ALL_API_URL = f"{ROOT_API}/datasets/get_all_stats/"
+JUDGMENT_DATASETS_EDIT_API_URL = f"{ROOT_API}/datasets/edit/"
 JUDGMENT_EVAL_LOG_API_URL = f"{ROOT_API}/log_eval_results/"
 JUDGMENT_EVAL_FETCH_API_URL = f"{ROOT_API}/fetch_eval_results/"
 JUDGMENT_EVAL_DELETE_API_URL = f"{ROOT_API}/delete_eval_results_by_project_and_run_name/"

{judgeval-0.0.12 → judgeval-0.0.13}/src/judgeval/data/datasets/dataset.py

@@ -162,7 +162,8 @@ class EvalDataset:
                 "additional_metadata": ast.literal_eval(row["additional_metadata"]) if pd.notna(row["additional_metadata"]) else dict(),
                 "tools_called": row["tools_called"].split(";") if pd.notna(row["tools_called"]) else [],
                 "expected_tools": row["expected_tools"].split(";") if pd.notna(row["expected_tools"]) else [],
-                "trace_id": row["trace_id"] if pd.notna(row["trace_id"]) else None
+                "trace_id": row["trace_id"] if pd.notna(row["trace_id"]) else None,
+                "example_id": str(row["example_id"]) if pd.notna(row["example_id"]) else None
             }
             if row["example"]:
                 data["name"] = row["name"] if pd.notna(row["name"]) else None

{judgeval-0.0.12 → judgeval-0.0.13}/src/judgeval/data/datasets/eval_dataset_client.py

@@ -1,5 +1,5 @@
 
-from typing import Optional
+from typing import Optional, List
 import requests
 from rich.progress import Progress, SpinnerColumn, TextColumn
 
@@ -7,7 +7,9 @@ from judgeval.common.logger import debug, error, warning, info
 from judgeval.constants import (
     JUDGMENT_DATASETS_PUSH_API_URL,
     JUDGMENT_DATASETS_PULL_API_URL, 
-    JUDGMENT_DATASETS_PULL_ALL_API_URL
+    JUDGMENT_DATASETS_PULL_ALL_API_URL,
+    JUDGMENT_DATASETS_EDIT_API_URL,
+    JUDGMENT_DATASETS_EXPORT_JSONL_API_URL
 )
 from judgeval.data import Example
 from judgeval.data.datasets import EvalDataset
@@ -23,7 +25,7 @@ class EvalDatasetClient:
     def create_dataset(self) -> EvalDataset:
         return EvalDataset(judgment_api_key=self.judgment_api_key)
     
-    def push(self, dataset: EvalDataset, alias: str,overwrite: Optional[bool] = False) -> bool:
+    def push(self, dataset: EvalDataset, alias: str, overwrite: Optional[bool] = False) -> bool:
         debug(f"Pushing dataset with alias '{alias}' (overwrite={overwrite})")
         if overwrite:
             warning(f"Overwrite enabled for alias '{alias}'")
@@ -56,12 +58,16 @@ class EvalDatasetClient:
                     "ground_truths": [g.to_dict() for g in dataset.ground_truths],
                     "examples": [e.to_dict() for e in dataset.examples],
                     "overwrite": overwrite,
-                    "judgment_api_key": dataset.judgment_api_key
+                    # "judgment_api_key": dataset.judgment_api_key
                 }
             try:
                 response = requests.post(
                     JUDGMENT_DATASETS_PUSH_API_URL, 
-                    json=content
+                    json=content,
+                    headers={
+                        "Content-Type": "application/json",
+                        "Authorization": f"Bearer {self.judgment_api_key}"
+                    }
                 )
                 if response.status_code == 500:
                     error(f"Server error during push: {content.get('message')}")
@@ -115,13 +121,17 @@ class EvalDatasetClient:
                 )
                 request_body = {
                     "alias": alias,
-                    "judgment_api_key": self.judgment_api_key
+                    # "judgment_api_key": self.judgment_api_key
                 }
 
                 try:
                     response = requests.post(
                         JUDGMENT_DATASETS_PULL_API_URL, 
-                        json=request_body
+                        json=request_body,
+                        headers={
+                            "Content-Type": "application/json",
+                            "Authorization": f"Bearer {self.judgment_api_key}"
+                        }
                     )
                     response.raise_for_status()
                 except requests.exceptions.RequestException as e:
@@ -169,13 +179,17 @@ class EvalDatasetClient:
                     total=100,
                 )
                 request_body = {
-                    "judgment_api_key": self.judgment_api_key
+                    # "judgment_api_key": self.judgment_api_key
                 }
 
                 try:
                     response = requests.post(
                         JUDGMENT_DATASETS_PULL_ALL_API_URL, 
-                        json=request_body
+                        json=request_body,
+                        headers={
+                            "Content-Type": "application/json",
+                            "Authorization": f"Bearer {self.judgment_api_key}"
+                        }
                     )
                     response.raise_for_status()
                 except requests.exceptions.RequestException as e:
@@ -191,3 +205,86 @@ class EvalDatasetClient:
                 )
 
                 return payload
+
+    def edit_dataset(self, alias: str, examples: List[Example], ground_truths: List[GroundTruthExample]) -> bool:
+        """
+        Edits the dataset on Judgment platform by adding new examples and ground truths
+
+        Mock request:
+        {
+            "alias": alias,
+            "examples": [...],
+            "ground_truths": [...],
+            "judgment_api_key": self.judgment_api_key
+        }
+        """
+        with Progress(
+                SpinnerColumn(style="rgb(106,0,255)"),
+                TextColumn("[progress.description]{task.description}"),
+                transient=False,
+            ) as progress:
+            task_id = progress.add_task(
+                f"Editing dataset [rgb(106,0,255)]'{alias}'[/rgb(106,0,255)] on Judgment...",
+                total=100,
+            )
+            
+            content = {
+                "alias": alias,
+                "examples": [e.to_dict() for e in examples],
+                "ground_truths": [g.to_dict() for g in ground_truths],
+                "judgment_api_key": self.judgment_api_key
+            }
+
+            try:
+                response = requests.post(
+                    JUDGMENT_DATASETS_EDIT_API_URL,
+                    json=content
+                )
+                response.raise_for_status()
+            except requests.exceptions.RequestException as e:
+                error(f"Error editing dataset: {str(e)}")
+                return False
+        
+            info(f"Successfully edited dataset '{alias}'")
+            return True
+
+    def export_jsonl(self, alias: str) -> requests.Response:
+        """Export dataset in JSONL format from Judgment platform"""
+        debug(f"Exporting dataset with alias '{alias}' as JSONL")
+        with Progress(
+            SpinnerColumn(style="rgb(106,0,255)"),
+            TextColumn("[progress.description]{task.description}"),
+            transient=False,
+        ) as progress:
+            task_id = progress.add_task(
+                f"Exporting [rgb(106,0,255)]'{alias}'[/rgb(106,0,255)] as JSONL...",
+                total=100,
+            )
+            try:
+                response = requests.post(
+                    JUDGMENT_DATASETS_EXPORT_JSONL_API_URL,
+                    json={"alias": alias},
+                    headers={
+                        "Content-Type": "application/json",
+                        "Authorization": f"Bearer {self.judgment_api_key}"
+                    },
+                    stream=True
+                )
+                response.raise_for_status()
+            except requests.exceptions.HTTPError as err:
+                if err.response.status_code == 404:
+                    error(f"Dataset not found: {alias}")
+                else:
+                    error(f"HTTP error during export: {err}")
+                raise
+            except Exception as e:
+                error(f"Error during export: {str(e)}")
+                raise
+                
+            info(f"Successfully exported dataset with alias '{alias}'")
+            progress.update(
+                task_id,
+                description=f"{progress.tasks[task_id].description} [rgb(25,227,160)]Done!)",
+            )
+            
+            return response

{judgeval-0.0.12 → judgeval-0.0.13}/src/judgeval/data/example.py

@@ -4,9 +4,11 @@ Classes for representing examples in a dataset.
 
 
 from typing import TypeVar, Optional, Any, Dict, List
-from pydantic import BaseModel
+from uuid import uuid4
+from pydantic import BaseModel, Field
 from enum import Enum
 from datetime import datetime
+import time
 
 
 Input = TypeVar('Input')
@@ -33,15 +35,19 @@ class Example(BaseModel):
     tools_called: Optional[List[str]] = None
     expected_tools: Optional[List[str]] = None
     name: Optional[str] = None
-    example_id: Optional[str] = None
+    example_id: str = Field(default_factory=lambda: str(uuid4()))
+    example_index: Optional[int] = None
     timestamp: Optional[str] = None
     trace_id: Optional[str] = None
 
     def __init__(self, **data):
-        super().__init__(**data)
+        if 'example_id' not in data:
+            data['example_id'] = str(uuid4())
         # Set timestamp if not provided
-        if self.timestamp is None:
-            self.timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+        if 'timestamp' not in data:
+            data['timestamp'] = datetime.now().strftime("%Y%m%d_%H%M%S")
+        super().__init__(**data)
+
 
     def to_dict(self):
         return {
@@ -55,6 +61,7 @@ class Example(BaseModel):
             "expected_tools": self.expected_tools,
             "name": self.name,
             "example_id": self.example_id,
+            "example_index": self.example_index,
             "timestamp": self.timestamp,
             "trace_id": self.trace_id
         }
@@ -71,6 +78,7 @@ class Example(BaseModel):
             f"expected_tools={self.expected_tools}, "
             f"name={self.name}, "
             f"example_id={self.example_id}, "
+            f"example_index={self.example_index}, "
             f"timestamp={self.timestamp}, "
             f"trace_id={self.trace_id})"
         )