judgeval 0.0.12__tar.gz → 0.0.14__tar.gz

{judgeval-0.0.12 → judgeval-0.0.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: judgeval
-Version: 0.0.12
+Version: 0.0.14
 Summary: Judgeval Package
 Project-URL: Homepage, https://github.com/JudgmentLabs/judgeval
 Project-URL: Issues, https://github.com/JudgmentLabs/judgeval/issues

{judgeval-0.0.12 → judgeval-0.0.14}/docs/api_reference/trace.mdx

@@ -12,27 +12,34 @@ The `Tracer` class is used to trace the execution of your LLM system.
 ```python
 from judgeval.common.tracer import Tracer
 
-tracer = Tracer()
+tracer = Tracer(project_name="my_project")
 ```
 
 <Note>
-The `Tracer` class is a singleton, so you only need to initialize it once in your application.
+The `Tracer` class is a singleton, so you only need to initialize it once in your application. 
+The `project_name` enables you to group traces by workflow, keeping all your evaluations and 
+observability tooling in one place.
 </Note>
 
-## Exporting traces
+## Explicitly exporting traces
 
 When using the `.trace()` context manager, you can control how your traces are exported to the Judgment platform by 
 providing the `project_name` argument. This allows you to group traces by workflow, keeping all your evaluations and 
 observability tooling in one place.
 
 ```python
-with tracer.trace("my_workflow", project_name="my_project"): 
+with tracer.trace(
+    name="my_workflow", 
+    project_name="my_project", 
+    overwrite=True
+    ) as trace: 
     ...
 ```
 
 `.trace()` has the following args:
 - `name`: The name of the trace. Can be make unique to each workflow run by using a timestamp or other unique identifier.
 - `project_name`: The name of the project to use for the trace. Used to group traces by workflow.
+- `overwrite`: Whether to overwrite the trace with the same `name` if it already exists.
 
 The `trace()` context manager yields a `TraceClient` object.
 

{judgeval-0.0.12 → judgeval-0.0.14}/docs/getting_started.mdx

@@ -32,7 +32,7 @@ large-scale evaluations. [Contact us](mailto:contact@judgmentlabs.ai) if you're
 sensitive data that has to reside in your private VPCs.
 </Note>
 
-# Create your first evaluation
+# Create Your First Evaluation
 
 ```python sample_eval.py
 from judgeval import JudgmentClient
@@ -68,6 +68,48 @@ is a scorer that checks if the output is hallucinated relative to the retrieved
 To learn more about using the Judgment Client to run evaluations, click [here](/api_reference/judgment_client).
 </Tip>
 
+# Create Your First Trace
+
+Beyond experimentation, `judgeval` supports monitoring your LLM systems in **production**. 
+Using our `tracing` module, you can **track your LLM system outputs from end to end**, allowing you to visualize the flow of your LLM system. 
+Additionally, you can **enable evaluations to run in real-time** using Judgment's state-of-the-art judge models. 
+
+```python trace_example.py
+from judgeval.common.tracer import Tracer, wrap
+from openai import OpenAI
+
+client = wrap(OpenAI())
+judgment = Tracer(project_name="my_project")
+
+@judgment.observe(span_type="tool")
+def my_tool():
+    return "Hello world!"
+
+@judgment.observe(span_type="function")
+def main():
+    res = client.chat.completions.create(
+        model="gpt-4o",
+        messages=[{"role": "user", "content": f"{my_tool()}"}]
+    )
+    return res.choices[0].message.content
+```
+
+
+<div style={{display: 'flex', justifyContent: 'center'}}>
+  ![Alt text](/images/trace_screenshot.png "Image of a RAG pipeline trace")
+</div>
+
+There are many benefits of monitoring your LLM systems in production with `judgeval`, including:
+- Detecting hallucinations and other quality issues **before they reach your customers**
+- Automatically creating experimental datasets from your **real-world production cases** for future improvement/optimization
+- Track and create alerts on **any metric** (e.g. latency, cost, hallucination, etc.)
+
+<Tip>
+To learn more about `judgeval`'s tracing module, click [here](/tracing/introduction).
+</Tip>
+
+
+
 # Create Your First Scorer
 `judgeval` offers three kinds of LLM scorers for your evaluation needs: ready-made, classifier scorers, and custom scorers.
 
@@ -264,22 +306,3 @@ A `Project` keeps track of `Evaluation Run`s in your project. Each `Evaluation R
 You can try different models (e.g. `gpt-4o`, `claude-3-5-sonnet`, etc.) and prompt templates in each `Evaluation Run` to find the 
 optimal setup for your LLM system. 
 </Tip>
-
-## Monitoring LLM Systems in Production
-
-Beyond experimenting and measuring historical performance, `judgeval` supports monitoring your LLM systems in **production**. 
-Using our `tracing` module, you can **track your LLM system outputs from end to end**, allowing you to visualize the flow of your LLM system. 
-Additionally, you can **enable evaluations to run in real-time** using Judgment's state-of-the-art judge models. 
-
-<div style={{display: 'flex', justifyContent: 'center'}}>
-  ![Alt text](/images/trace_screenshot.png "Image of a RAG pipeline trace")
-</div>
-
-There are many benefits of monitoring your LLM systems in production with `judgeval`, including:
-- Detecting hallucinations and other quality issues **before they reach your customers**
-- Automatically creating experimental datasets from your **real-world production cases** for future improvement/optimization
-- Track and create alerts on **any metric** (e.g. latency, cost, hallucination, etc.)
-
-<Tip>
-To learn more about `judgeval`'s tracing module, click [here](/tracing/introduction).
-</Tip>

{judgeval-0.0.12 → judgeval-0.0.14}/docs/monitoring/tracing.mdx

@@ -18,24 +18,25 @@ Using tracing, you can:
 
 ## Tracing Your Workflow ##
 
-Setting up tracing with `judgeval` takes three simple steps:
+Setting up tracing with `judgeval` takes two simple steps:
 
-### 1. Initialize a tracer with your API key
+### 1. Initialize a tracer with your API key and project name
 
 ```python
 from judgeval.common.tracer import Tracer
 
-judgment = Tracer()  # loads from JUDGMENT_API_KEY env var
+judgment = Tracer(project_name="my_project")  # loads from JUDGMENT_API_KEY env var
 ```
 
 <Note>
-    The [Judgment tracer](/api_reference/trace) is a singleton object that should be shared across your application.
+    The [Judgment tracer](/api_reference/trace) is a singleton object that should be shared across your application. 
+    Your project name will be used to organize your traces in one place on the Judgment platform.
 </Note>
 
 
 ### 2. Wrap your workflow components
 
-`judgeval` provides three wrapping mechanisms for your workflow components:
+`judgeval` provides wrapping mechanisms for your workflow components:
 
 #### `wrap()` ####
 The `wrap()` function goes over your LLM client (e.g. OpenAI, Anthropic, etc.) and captures metadata surrounding your LLM calls, such as:
@@ -44,6 +45,14 @@ The `wrap()` function goes over your LLM client (e.g. OpenAI, Anthropic, etc.) a
 - Prompt/Completion
 - Model name
 
+Here's an example of using `wrap()` on an OpenAI client:
+```python
+from openai import OpenAI
+from judgeval.common.tracer import wrap
+
+client = wrap(OpenAI())
+```
+
 #### `@observe` ####
 The `@observe` decorator wraps your functions/tools and captures metadata surrounding your function calls, such as:
 - Latency
@@ -63,30 +72,20 @@ def my_tool():
 ```
 
 <Note>
-    The `@observe` decorator is used on top of helper functions that you write, but is not designed to be used 
-    on your "main" function. For more information, see the `context manager` section below.
+    `span_type` is a string that you can use to categorize and organize your trace spans. 
+    Span types are displayed on the trace UI to easily nagivate a visualization of your workflow. 
+    Common span types include `tool`, `function`, `retriever`, `database`, `web search`, etc.
 </Note>
 
-#### `context manager` ####
-
-In your main function (e.g. the one that executes the primary workflow logic), you can use the `with judgment.trace()` context manager to trace the entire workflow.
-
-The context manager can **save/print the state of the trace at any point in the workflow**.
-This is useful for debugging or exporting any state of your workflow to run an evaluation from!
-
-<Tip>
-    The `with judgment.trace()` context manager detects any `@observe` decorated functions or wrapped LLM calls within the context and automatically captures their metadata.
-</Tip>
-
 
 #### Putting it all Together
-Here's a complete example of using the `with judgment.trace()` context manager with the other tracing mechanisms:
+Here's a complete example of using judgeval's tracing mechanisms:
 ```python
 from judgeval.common.tracer import Tracer, wrap
 from openai import OpenAI
 
 openai_client = wrap(OpenAI())
-judgment = Tracer()  # loads from JUDGMENT_API_KEY env var
+judgment = Tracer(project_name="my_project")  # loads from JUDGMENT_API_KEY env var
 
 @judgment.observe(span_type="tool")
 def my_tool():
@@ -101,28 +100,10 @@ def my_llm_call():
     )
     return res.choices[0].message.content
 
+@judgment.observe(span_type="function")
 def main():
-    with judgment.trace(
-        "main_workflow", 
-        project_name="my_project"
-    ) as trace:
-        res = my_llm_call()
-        trace.save()
-        trace.print()
-        return res
-```
-
-The printed trace appears as follows on the terminal:
-```
-→ main_workflow (trace: main_workflow)
-  → my_llm_call (trace: my_llm_call)
-    Input: {'args': [], 'kwargs': {}}
-    → my_tool (trace: my_tool)
-      Input: {'args': [], 'kwargs': {}}
-      Output: Hello world!
-    ← my_tool (0.000s)
-    Output: Hello! How can I assist you today?
-  ← my_llm_call (0.789s)
+    res = my_llm_call()
+    return res
 ```
 
 And the trace will appear on the Judgment platform as follows:
@@ -142,32 +123,27 @@ To execute an asynchronous evaluation, you can use the `trace.async_evaluate()`
 
 ```python
 from judgeval.common.tracer import Tracer
-from judgeval.scorers import FaithfulnessScorer
+from judgeval.scorers import AnswerRelevancyScorer
 
-judgment = Tracer()
+judgment = Tracer(project_name="my_project")
 
+@judgment.observe(span_type="function")
 def main():
-    with judgment.trace(
-        "main_workflow", 
-        project_name="my_project"
-    ) as trace:
-        retrieved_info = ...   # from knowledge base
-        res = ...  # your main workflow logic
-        
-        judgment.get_current_trace().async_evaluate(
-            scorers=[FaithfulnesssScorer(threshold=0.5)],
-            input="",
-            actual_output=res,
-            retrieval_context=[retrieved_info],
-            model="gpt-4o-mini",
-        )
-        return res
+    query = "What is the capital of France?"
+    res = ...  # Your workflow logic
+    
+    judgment.get_current_trace().async_evaluate(
+        scorers=[AnswerRelevancyScorer(threshold=0.5)],
+        input="",
+        actual_output=res,
+        model="gpt-4o",
+    )
+    return res
 ```
 
 <Tip>
-You can organize how your async evaluation runs are logged to the Judgment platform by using the 
-`project_name` argument in the `trace` context manager. See our [API documentation](/api_reference/trace) 
-for more information.
+    Your async evaluations will be logged to the Judgment platform as part of the original trace and 
+    a new evaluation will be created on the Judgment platform.
 </Tip>
 
 ## Example: OpenAI Travel Agent
@@ -183,4 +159,53 @@ In this video, we'll walk through all of the topics covered in this guide by tra
     allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" 
     referrerpolicy="strict-origin-when-cross-origin" 
     allowfullscreen
-></iframe>
+></iframe>
+
+
+## Advanced: Customizing Traces Using the Context Manager ##
+
+If you need to customize your tracing context, you can use the `with judgment.trace()` context manager.
+
+The context manager can **save/print the state of the trace at any point in the workflow**.
+This is useful for debugging or exporting any state of your workflow to run an evaluation from!
+
+<Tip>
+    The `with judgment.trace()` context manager detects any `@observe` decorated functions or wrapped LLM calls within the context and automatically captures their metadata.
+</Tip>
+
+Here's an example of using the context manager to trace a workflow:
+```python
+from judgeval.common.tracer import Tracer, wrap
+from openai import OpenAI
+
+judgment = Tracer(project_name="my_project")
+client = wrap(OpenAI())
+
+@judgment.observe(span_type="tool")
+def my_tool():
+    return "Hello world!"
+
+def main():
+    with judgment.trace(name="my_workflow") as trace:
+        res = client.chat.completions.create(
+            model="gpt-4o",
+            messages=[{"role": "user", "content": f"{my_tool()}"}]
+        )
+    
+    trace.print()  # prints the state of the trace to console
+    trace.save()  # saves the current state of the trace to the Judgment platform
+
+    return res.choices[0].message.content
+```
+
+<Warning>
+    The `with judgment.trace()` context manager should only be used if you need to customize the context 
+    over which you're tracing. In most cases, you should trace using the `@observe` decorator.
+</Warning>
+
+
+
+
+
+
+

{judgeval-0.0.12 → judgeval-0.0.14}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "judgeval"
-version = "0.0.12"
+version = "0.0.14"
 authors = [
     { name="Andrew Li", email="andrew@judgmentlabs.ai" },
     { name="Alex Shan", email="alex@judgmentlabs.ai" },

{judgeval-0.0.12 → judgeval-0.0.14}/src/demo/cookbooks/openai_travel_agent/agent.py

@@ -50,7 +50,7 @@ async def get_flights(destination):
     judgment.get_current_trace().async_evaluate(
         scorers=[AnswerRelevancyScorer(threshold=0.5)],
         input=prompt,
-        actual_output=flights_search,
+        actual_output=flights_search["results"],
         model="gpt-4",
     )
     return flights_search
@@ -63,7 +63,7 @@ async def get_weather(destination, start_date, end_date):
     judgment.get_current_trace().async_evaluate(
         scorers=[AnswerRelevancyScorer(threshold=0.5)],
         input=prompt,
-        actual_output=weather_search,
+        actual_output=weather_search["results"],
         model="gpt-4",
     )
     return weather_search

{judgeval-0.0.12 → judgeval-0.0.14}/src/judgeval/common/tracer.py

@@ -188,8 +188,9 @@ class TraceManagerClient:
     - Saving a trace
     - Deleting a trace
     """
-    def __init__(self, judgment_api_key: str):
+    def __init__(self, judgment_api_key: str, organization_id: str):
         self.judgment_api_key = judgment_api_key
+        self.organization_id = organization_id
 
     def fetch_trace(self, trace_id: str):
         """
@@ -199,10 +200,11 @@ class TraceManagerClient:
             JUDGMENT_TRACES_FETCH_API_URL,
             json={
                 "trace_id": trace_id,
-                "judgment_api_key": self.judgment_api_key,
             },
             headers={
                 "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.judgment_api_key}",
+                "X-Organization-Id": self.organization_id
             }
         )
 
@@ -225,6 +227,8 @@ class TraceManagerClient:
             json=trace_data,
             headers={
                 "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.judgment_api_key}",
+                "X-Organization-Id": self.organization_id
             }
         )
         
@@ -243,11 +247,12 @@ class TraceManagerClient:
         response = requests.delete(
             JUDGMENT_TRACES_DELETE_API_URL,
             json={
-                "judgment_api_key": self.judgment_api_key,
                 "trace_ids": [trace_id],
             },
             headers={
                 "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.judgment_api_key}",
+                "X-Organization-Id": self.organization_id
             }
         )
 
@@ -263,11 +268,12 @@ class TraceManagerClient:
         response = requests.delete(
             JUDGMENT_TRACES_DELETE_API_URL,
             json={
-                "judgment_api_key": self.judgment_api_key,
                 "trace_ids": trace_ids,
             },
             headers={
                 "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.judgment_api_key}",
+                "X-Organization-Id": self.organization_id
             }
         )
 
@@ -290,7 +296,7 @@ class TraceClient:
         self.span_type = None
         self._current_span: Optional[TraceEntry] = None
         self.overwrite = overwrite
-        self.trace_manager_client = TraceManagerClient(tracer.api_key)  # Manages DB operations for trace data
+        self.trace_manager_client = TraceManagerClient(tracer.api_key, tracer.organization_id)  # Manages DB operations for trace data
         
     @contextmanager
     def span(self, name: str, span_type: SpanType = "span"):
@@ -367,6 +373,7 @@ class TraceClient:
             raise ValueError(f"Failed to load scorers: {str(e)}")
         
         eval_run = EvaluationRun(
+            organization_id=self.tracer.organization_id,
             log_results=log_results,
             project_name=self.project_name,
             eval_name=f"{self.name.capitalize()}-"
@@ -542,7 +549,6 @@ class TraceClient:
         # Create trace document
         trace_data = {
             "trace_id": self.trace_id,
-            "api_key": self.tracer.api_key,
             "name": self.name,
             "project_name": self.project_name,
             "created_at": datetime.fromtimestamp(self.start_time).isoformat(),
@@ -564,6 +570,8 @@ class TraceClient:
             channel = connection.channel()
             
             channel.queue_declare(queue=RABBITMQ_QUEUE, durable=True)
+            trace_data["judgment_api_key"] = self.tracer.api_key
+            trace_data["organization_id"] = self.tracer.organization_id
             
             channel.basic_publish(
                 exchange='',
@@ -589,14 +597,18 @@ class Tracer:
             cls._instance = super(Tracer, cls).__new__(cls)
         return cls._instance
 
-    def __init__(self, api_key: str = os.getenv("JUDGMENT_API_KEY"), project_name: str = "default_project"):
+    def __init__(self, api_key: str = os.getenv("JUDGMENT_API_KEY"), project_name: str = "default_project", organization_id: str = os.getenv("ORGANIZATION_ID")):
         if not hasattr(self, 'initialized'):
             if not api_key:
                 raise ValueError("Tracer must be configured with a Judgment API key")
             
+            if not organization_id:
+                raise ValueError("Tracer must be configured with an Organization ID")
+            
             self.api_key: str = api_key
             self.project_name: str = project_name
             self.client: JudgmentClient = JudgmentClient(judgment_api_key=api_key)
+            self.organization_id: str = organization_id
             self.depth: int = 0
             self._current_trace: Optional[str] = None
             self.initialized: bool = True

{judgeval-0.0.12 → judgeval-0.0.14}/src/judgeval/constants.py

@@ -36,7 +36,9 @@ ROOT_API = os.getenv("JUDGMENT_API_URL", "https://api.judgmentlabs.ai")
 JUDGMENT_EVAL_API_URL = f"{ROOT_API}/evaluate/"
 JUDGMENT_DATASETS_PUSH_API_URL = f"{ROOT_API}/datasets/push/"
 JUDGMENT_DATASETS_PULL_API_URL = f"{ROOT_API}/datasets/pull/"
+JUDGMENT_DATASETS_EXPORT_JSONL_API_URL = f"{ROOT_API}/datasets/export_jsonl/"
 JUDGMENT_DATASETS_PULL_ALL_API_URL = f"{ROOT_API}/datasets/get_all_stats/"
+JUDGMENT_DATASETS_EDIT_API_URL = f"{ROOT_API}/datasets/edit/"
 JUDGMENT_EVAL_LOG_API_URL = f"{ROOT_API}/log_eval_results/"
 JUDGMENT_EVAL_FETCH_API_URL = f"{ROOT_API}/fetch_eval_results/"
 JUDGMENT_EVAL_DELETE_API_URL = f"{ROOT_API}/delete_eval_results_by_project_and_run_name/"

{judgeval-0.0.12 → judgeval-0.0.14}/src/judgeval/data/datasets/dataset.py

@@ -17,9 +17,10 @@ class EvalDataset:
     _alias: Union[str, None] = field(default=None)
     _id: Union[str, None] = field(default=None)
     judgment_api_key: str = field(default="")
-
+    organization_id: str = field(default="")
     def __init__(self, 
                  judgment_api_key: str = os.getenv("JUDGMENT_API_KEY"),  
+                 organization_id: str = os.getenv("ORGANIZATION_ID"),
                  ground_truths: List[GroundTruthExample] = [], 
                  examples: List[Example] = [],
                  ):
@@ -31,7 +32,7 @@ class EvalDataset:
         self._alias = None
         self._id = None
         self.judgment_api_key = judgment_api_key
-        
+        self.organization_id = organization_id
 
     def add_from_json(self, file_path: str) -> None:
         debug(f"Loading dataset from JSON file: {file_path}")
@@ -162,7 +163,8 @@ class EvalDataset:
                 "additional_metadata": ast.literal_eval(row["additional_metadata"]) if pd.notna(row["additional_metadata"]) else dict(),
                 "tools_called": row["tools_called"].split(";") if pd.notna(row["tools_called"]) else [],
                 "expected_tools": row["expected_tools"].split(";") if pd.notna(row["expected_tools"]) else [],
-                "trace_id": row["trace_id"] if pd.notna(row["trace_id"]) else None
+                "trace_id": row["trace_id"] if pd.notna(row["trace_id"]) else None,
+                "example_id": str(row["example_id"]) if pd.notna(row["example_id"]) else None
             }
             if row["example"]:
                 data["name"] = row["name"] if pd.notna(row["name"]) else None