judgeval 0.0.26__tar.gz → 0.0.27__tar.gz

{judgeval-0.0.26 → judgeval-0.0.27}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: judgeval
-Version: 0.0.26
+Version: 0.0.27
 Summary: Judgeval Package
 Project-URL: Homepage, https://github.com/JudgmentLabs/judgeval
 Project-URL: Issues, https://github.com/JudgmentLabs/judgeval/issues

{judgeval-0.0.26 → judgeval-0.0.27}/docs/evaluation/scorers/custom_scorers.mdx

@@ -4,6 +4,7 @@ description: ""
 ---
 
 If none of `judgeval`'s built-in scorers fit your evaluation criteria, you can easily build your own custom metric to be run through a `JudgevalScorer`. 
+
 `JudgevalScorer`s are **automatically integrated** within `judgeval`'s infrastructure, so you can:
 - Run your own scorer with the same syntax as any other `judgeval` scorer.
 - Use `judgeval`'s batched evaluation infrastructure to execute **scalable evaluation runs**.
@@ -78,7 +79,6 @@ You can optionally set the self.reason attribute, depending on your preference.
 </Note>
 
 These methods are the core of your scorer, and you can implement them in any way you want. **Be creative!** 
-Check out this list of examples our users have implemented if you need inspiration: TODO add link here
 
 #### Handling Errors
 If you want to handle errors gracefully, you can use a `try` block and in the `except` block, set the `self.error` attribute to the error message. 
@@ -144,11 +144,37 @@ class SampleScorer(JudgevalScorer):
     def __name__(self):
         return "Sample Scorer"
 ```
-
 **Congratulations!** 🎉 
 
 You've made your first custom judgeval scorer! Now that your scorer is implemented, you can run it on your own datasets 
 just like any other `judgeval` scorer. Your scorer is fully integrated with `judgeval`'s infrastructure so you can view it on 
 the [Judgment platform](/judgment/introduction) too.
 
-For more examples, check out some of the custom scorers our users have implemented: TODO add link here.
+## Using a Custom Scorer
+
+Once you've implemented your custom scorer, you can use it in the same way as any other scorer in `judgeval`. 
+They can be run in conjunction with other scorers in a single evaluation run!
+
+```python run_custom_scorer.py
+from judgeval import JudgmentClient
+from your_custom_scorer import SampleScorer
+
+client = JudgmentClient()
+sample_scorer = SampleScorer()
+
+results = client.run_evaluation(
+    examples=[example1],
+    scorers=[sample_scorer],
+    model="gpt-4o"
+)
+```
+
+## Real World Examples
+
+You can find some real world examples of how our community has used custom `JudgevalScorer`s to evaluate their LLM systems in our [cookbook repository](https://github.com/JudgmentLabs/judgment-cookbook/tree/main/cookbooks/custom_scorers)! 
+Here are some of our favorites:
+
+- [Code Style Scorer](https://github.com/JudgmentLabs/judgment-cookbook/blob/main/cookbooks/custom_scorers/code_style_scorer.py) - Evaluates code quality and style
+- [Cold Email Scorer](https://github.com/JudgmentLabs/judgment-cookbook/blob/main/cookbooks/custom_scorers/cold_email_scorer.py) - Evaluates the effectiveness of cold emails
+
+For more examples and detailed documentation on custom scorers, check out our [Custom Scorers Cookbook](https://github.com/JudgmentLabs/judgment-cookbook/blob/main/cookbooks/custom_scorers/README.md).

{judgeval-0.0.26 → judgeval-0.0.27}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "judgeval"
-version = "0.0.26"
+version = "0.0.27"
 authors = [
     { name="Andrew Li", email="andrew@judgmentlabs.ai" },
     { name="Alex Shan", email="alex@judgmentlabs.ai" },

judgeval-0.0.27/src/demo/new_trace/example_complex_async.py

@@ -0,0 +1,232 @@
+import asyncio
+import time
+import sys
+import os
+import functools
+from unittest.mock import MagicMock, patch
+from typing import Dict, Optional, List
+import uuid
+import json
+
+# Standard library imports needed for the new class
+import concurrent.futures
+import contextvars
+# Needed for partial in the executor
+
+# Add src directory to Python path for imports
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src'))
+
+# Import and mock necessary components before initializing the tracer
+from judgeval.common.tracer import Tracer, JudgmentClient, TraceClient, current_trace_var, TraceEntry, TraceManagerClient, TraceThreadPoolExecutor # Import the new class
+
+# Initialize the tracer with test values
+tracer = Tracer(
+    project_name="complex_async_test"
+)
+
+# In this example, we'll use a single trace with spans for all function calls
+@tracer.observe(name="root_function")
+async def root_function():
+    print("Root function starting")
+    
+    # Direct await call to level 2
+    result1 = await level2_function("direct")
+    
+    # Parallel calls (gather) to level 2 functions
+    # These should be level 2 - direct children of root
+    # Create two truly parallel functions that both have root_function as parent
+    level2_parallel1_task = level2_parallel1("gather1")
+    level2_parallel2_task = level2_parallel2("gather2")
+    
+    # Use trace_gather instead of asyncio.gather to preserve context
+    # This ensures parent-child relationships are maintained in parallel tasks
+    # result2, result3 = await trace_gather(level2_parallel1_task, level2_parallel2_task) # OLD
+    result2, result3 = await asyncio.gather(level2_parallel1_task, level2_parallel2_task) # Use standard gather
+    
+    
+    print("Root function completed")
+    return f"Root results: {result1}, {result2}, {result3}"
+
+# Level 2 - Direct child of root
+# Using observe with same tracer - this will create spans in the parent trace
+@tracer.observe()
+async def level2_function(param):
+    # Capture this function in a span within the current trace
+    print(f"Level 2 function with {param}")
+
+    # Call to level 3
+    result = await level3_function(f"{param}_child")
+
+    return f"level2:{result}"
+
+# Level 2 - First parallel function
+@tracer.observe()
+async def level2_parallel1(param):
+    # Capture this function in a span within the current trace
+    print(f"Level 2 parallel 1 with {param}")
+
+    # This parallel function makes another parallel call to level 3 functions
+    # These should be direct children of level2_parallel1
+    # r1, r2 = await trace_gather( # OLD
+    r1, r2 = await asyncio.gather( # Use standard gather
+        level3_parallel1(f"{param}_1"),
+        level3_parallel2(f"{param}_2")
+    )
+
+    return f"level2_parallel1:{r1},{r2}"
+
+# Level 2 - Second parallel function
+@tracer.observe()
+async def level2_parallel2(param):
+    # Capture this function in a span within the current trace
+    print(f"Level 2 parallel 2 with {param}")
+
+    # Direct await to level 3
+    result = await level3_function(f"{param}_direct")
+
+    return f"level2_parallel2:{result}"
+
+# Level 3 - Child of level 2 direct
+@tracer.observe()
+async def level3_function(param):
+    # Capture this function in a span within the current trace
+    print(f"Level 3 function with {param}")
+
+    # Call to level 4
+    result = await level4_function(f"{param}_deep")
+
+    return f"level3:{result}"
+
+# Level 3 - First parallel function called by level2_parallel1
+@tracer.observe()
+async def level3_parallel1(param):
+    # Capture this function in a span within the current trace
+    print(f"Level 3 parallel 1 with {param}")
+
+    # This makes a nested gather call with level 4 functions
+    # results = await trace_gather( # OLD
+    results = await asyncio.gather( # Use standard gather
+        level4_function(f"{param}_a"),
+        level4_function(f"{param}_b"),
+        level4_function(f"{param}_c")
+    )
+
+    return f"level3_p1:{','.join(results)}"
+
+# Level 3 - Second parallel function called by level2_parallel1
+@tracer.observe()
+async def level3_parallel2(param):
+    # Capture this function in a span within the current trace
+    print(f"Level 3 parallel 2 with {param}")
+    await asyncio.sleep(0.1)
+
+    # Direct call to level 4
+    result = await level4_deep_function(f"{param}_deep")
+
+    return f"level3_p2:{result}"
+
+# Level 4 - Deepest regular function
+@tracer.observe()
+async def level4_function(param):
+    # Capture this function in a span within the current trace
+    print(f"Level 4 function with {param}")
+    await asyncio.sleep(0.05)
+
+    return f"level4:{param}"
+
+# Level 4 - Deep function that calls level 5
+@tracer.observe()
+async def level4_deep_function(param):
+    # Capture this function in a span within the current trace
+    print(f"Level 4 deep function with {param}")
+
+    # Call to level 5 (maximum depth)
+    result = await level5_function(f"{param}_final")
+    test = await fib(5)
+    return f"level4_deep:{result}"
+
+@tracer.observe()
+async def fib(n):
+    if n <= 1:
+        return n
+    return await fib(n-1) + await fib(n-2)
+
+# Level 5 - Deepest level
+@tracer.observe()
+async def level5_function(param):
+    # Capture this function in a span within the current trace
+    print(f"Level 5 function with {param}")
+    await asyncio.sleep(0.05)
+
+    return f"level5:{param}"
+    
+# --- Synchronous ThreadPoolExecutor Test ---
+
+@tracer.observe(name="sync_child_task1")
+def sync_child_task1(param):
+    """A simple synchronous function to be run in a thread."""
+    print(f"SYNC CHILD 1: Received {param}. Sleeping...")
+    time.sleep(0.15)
+    result = f"Result from sync_child_task1 with {param}"
+    print("SYNC CHILD 1: Done.")
+    return result
+
+@tracer.observe(name="sync_child_task2")
+def sync_child_task2(param1, param2):
+    """Another simple synchronous function."""
+    print(f"SYNC CHILD 2: Received {param1} and {param2}. Sleeping...")
+    time.sleep(0.05)
+    result = f"Result from sync_child_task2 with {param1}, {param2}"
+    print("SYNC CHILD 2: Done.")
+    return result
+
+@tracer.observe(name="sync_parent_func")
+def sync_parent_func():
+    """This function uses TraceThreadPoolExecutor to run sync tasks."""
+    print("SYNC PARENT: Starting...")
+    results = []
+    # Use the TraceThreadPoolExecutor instead of the standard one
+    with TraceThreadPoolExecutor(max_workers=2) as executor:
+        print("SYNC PARENT: Submitting tasks to TraceThreadPoolExecutor...")
+        future1 = executor.submit(sync_child_task1, "data_for_task1")
+        future2 = executor.submit(sync_child_task2, "data1_for_task2", "data2_for_task2")
+
+        print("SYNC PARENT: Waiting for futures...")
+        # Wait for futures and collect results (demonstrates typical usage)
+        for future in concurrent.futures.as_completed([future1, future2]):
+            try:
+                results.append(future.result())
+            except Exception as exc:
+                print(f"SYNC PARENT: Generated an exception: {exc}")
+                results.append(f"Error: {exc}")
+
+    print("SYNC PARENT: Finished.")
+    return results
+
+# --- End Synchronous Test ---
+
+async def main():
+    # Run the root function which has deep nesting and nested parallel calls
+    start_time = time.time()
+    result_async = await root_function()
+    end_time = time.time()
+    print(f"\nAsync Final result: {result_async}")
+    print(f"Async Total execution time: {end_time - start_time:.2f} seconds")
+    
+    print("\n" + "="*20 + " Starting Sync ThreadPool Test " + "="*20 + "\n")
+
+    # --- Run the synchronous thread pool test ---
+    # Note: We run this *outside* the async root_function's trace
+    # If we wanted it nested, we'd need @tracer.observe on main or call it from root_function
+    # For simplicity, let's trace it separately by calling it directly.
+    # The @tracer.observe on sync_parent_func will create its own root trace.
+    start_time_sync = time.time()
+    result_sync = sync_parent_func() # This will be traced automatically
+    end_time_sync = time.time()
+    print(f"\nSync Final results: {result_sync}")
+    print(f"Sync Total execution time: {end_time_sync - start_time_sync:.2f} seconds")
+    # --- End synchronous test call ---
+
+if __name__ == "__main__":
+    # Run the complex async example
+    asyncio.run(main()) 

{judgeval-0.0.26 → judgeval-0.0.27}/src/demo/travel_agent.py

@@ -84,7 +84,7 @@ Key Information:
 ]
 
 client = wrap(openai.Client(api_key=os.getenv("OPENAI_API_KEY")))
-judgment = Tracer(api_key=os.getenv("JUDGMENT_API_KEY"), project_name="travel_agent_demo")
+judgment = Tracer(api_key=os.getenv("JUDGMENT_API_KEY"), project_name="travel_agent_demo", enable_evaluations=False, enable_monitoring=False)
 
 def populate_vector_db(collection, destinations_data):
     """