speedy-utils 1.1.23__tar.gz → 1.1.24__tar.gz

speedy_utils-1.1.24/IMPROVEMENTS.md

@@ -0,0 +1,141 @@
+# Multi-thread Error Tracing Improvements
+
+## Summary
+
+Significantly improved error tracing in `multi_thread` to focus on user code rather than infrastructure frames, making debugging much faster and easier.
+
+## Problem
+
+Previously, when errors occurred in functions executed by `multi_thread`, the traceback was cluttered with infrastructure frames:
+
+- `concurrent.futures` internals
+- `threading.py` frames
+- `multi_worker/thread.py` infrastructure code
+
+This made it difficult to quickly identify the actual problem in user code.
+
+### Example of OLD behavior:
+
+```
+TypeError                                 Traceback (most recent call last)
+Cell In[810], line 35
+     33     choices = multi_thread(fns, range(n))
+     34     return choices
+---> 35 choices = translate()
+
+File ~/projects/speedy_utils/src/speedy_utils/multi_worker/thread.py:474, in multi_thread(...)
+    472 idx, logical_size = _future_meta(fut)
+    473 try:
+--> 474     result = fut.result()
+    475 except Exception as exc:
+    476     if stop_on_error:
+
+File ~/.local/share/uv/python/.../concurrent/futures/_base.py:449, in Future.result(...)
+    447     raise CancelledError()
+    448 elif self._state == FINISHED:
+--> 449     return self.__get_result()
+
+... (many more infrastructure frames) ...
+
+TypeError: 'list' object is not callable
+```
+
+## Solution
+
+### 1. Added `UserFunctionError` Exception Class
+
+A custom exception wrapper that:
+
+- Captures the original exception
+- Stores the function name and input that caused the error
+- Filters traceback to include only user code frames
+- Provides clear, focused error messages
+
+### 2. Enhanced `_worker` Function
+
+- Added validation to detect common mistakes (e.g., passing a list instead of a function)
+- Filters tracebacks to remove infrastructure frames
+- Wraps user function errors in `UserFunctionError` with clean context
+- Provides helpful hints for common mistakes
+
+### 3. Improved Error Reporting in `multi_thread`
+
+- Logs clear error messages showing function name and input
+- Displays only user code in tracebacks
+- Re-raises exceptions with cleaned messages
+- Maintains proper exception chaining while hiding infrastructure noise
+
+## Benefits
+
+### Clear Error Messages
+
+```
+Error in function "process_item" with input: 0
+
+User code traceback:
+  File "your_script.py", line 20, in process_item
+    return my_list(x)
+           ^^^^^^^^^^
+TypeError: 'list' object is not callable
+```
+
+### Helpful Hints
+
+```
+TypeError:
+multi_thread: func parameter must be callable, got list: [...]
+Hint: Did you accidentally pass a list instead of a function?
+```
+
+### Nested Function Support
+
+Shows complete call chain through user code:
+
+```
+Error in function "process_data" with input: 0
+
+User code traceback:
+  File "your_script.py", line 44, in process_data
+    return validate_and_calc(val)
+           ^^^^^^^^^^^^^^^^^^^^^^
+  File "your_script.py", line 42, in validate_and_calc
+    return 100 / x
+           ~~~~^~~
+ZeroDivisionError: division by zero
+```
+
+## Key Improvements
+
+✅ **Errors show function name and problematic input**
+✅ **Tracebacks filtered to show only user code**
+✅ **No concurrent.futures/threading clutter**
+✅ **Helpful hints for common mistakes**
+✅ **Clear, actionable error messages**
+✅ **Maintains backward compatibility - all existing tests pass**
+
+## Testing
+
+Run the comprehensive demo to see all improvements:
+
+```bash
+python tests/test_multithread_error_trace.py
+```
+
+This demonstrates:
+
+1. Simple function errors
+2. Nested function call traces
+3. Common parameter type mistakes
+4. Various exception types (TypeError, ValueError, AttributeError, etc.)
+
+## Code Changes
+
+Main files modified:
+
+- `src/speedy_utils/multi_worker/thread.py`:
+  - Added `UserFunctionError` exception class
+  - Enhanced `_worker` function with validation and error filtering
+  - Improved error handling in `multi_thread` main loop
+  - Added imports for `sys` and `traceback`
+
+All changes maintain backward compatibility - existing code continues to work unchanged.

{speedy_utils-1.1.23 → speedy_utils-1.1.24}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: speedy-utils
-Version: 1.1.23
+Version: 1.1.24
 Summary: Fast and easy-to-use package for data science
 Project-URL: Homepage, https://github.com/anhvth/speedy
 Project-URL: Repository, https://github.com/anhvth/speedy

speedy_utils-1.1.24/examples/temperature_range_example.py

@@ -0,0 +1,119 @@
+"""Example demonstrating temperature range sampling with LLM."""
+
+from llm_utils import LLM
+from pydantic import BaseModel
+
+
+class CreativeStory(BaseModel):
+    """A creative story output."""
+
+    title: str
+    story: str
+    moral: str
+
+
+def example_temperature_range_text():
+    """Example: Sample text responses with different temperatures."""
+    print("=" * 60)
+    print("Example 1: Temperature Range Sampling (Text Completion)")
+    print("=" * 60)
+
+    llm = LLM(
+        instruction="You are a creative writer. Write a very short story.",
+        output_model=str,
+    )
+
+    prompt = "Write a one-sentence story about a brave mouse."
+
+    # Sample with 5 different temperatures from 0.1 to 1.0
+    responses = llm(
+        prompt,
+        temperature_ranges=(0.1, 1.0),
+        n=5,
+    )
+
+    print(f"\nGenerated {len(responses)} responses with varying temperatures:\n")
+    for i, resp in enumerate(responses):
+        temp = 0.1 + i * ((1.0 - 0.1) / (5 - 1))
+        print(f"Temperature ~{temp:.2f}:")
+        print(f"  {resp['parsed']}\n")
+
+
+def example_temperature_range_pydantic():
+    """Example: Sample structured responses with different temperatures."""
+    print("=" * 60)
+    print("Example 2: Temperature Range with Pydantic Models")
+    print("=" * 60)
+
+    llm = LLM(
+        instruction="Create a creative short story with a moral lesson.",
+        output_model=CreativeStory,
+    )
+
+    prompt = "Topic: A robot learning to feel emotions"
+
+    # Sample with 3 different temperatures from 0.5 to 1.5
+    responses = llm(
+        prompt,
+        temperature_ranges=(0.5, 1.5),
+        n=3,
+    )
+
+    print(f"\nGenerated {len(responses)} stories with varying creativity:\n")
+    for i, resp in enumerate(responses):
+        temp = 0.5 + i * ((1.5 - 0.5) / (3 - 1))
+        story = resp["parsed"]
+        print(f"Temperature ~{temp:.2f}:")
+        print(f"  Title: {story.title}")
+        print(f"  Story: {story.story[:80]}...")
+        print(f"  Moral: {story.moral}\n")
+
+
+def example_two_step_parsing():
+    """Example: Two-step Pydantic parsing for models with reasoning."""
+    print("=" * 60)
+    print("Example 3: Two-Step Pydantic Parsing")
+    print("=" * 60)
+
+    llm = LLM(
+        instruction=("Analyze the given text and extract structured information. Think through your analysis first."),
+        output_model=CreativeStory,
+    )
+
+    prompt = "Analyze the story: 'The tortoise won the race by persistence.'"
+
+    # Use two-step parsing (useful for reasoning models)
+    response = llm(
+        prompt,
+        two_step_parse_pydantic=True,
+    )[0]
+
+    story = response["parsed"]
+    print("\nExtracted structure:")
+    print(f"  Title: {story.title}")
+    print(f"  Story: {story.story}")
+    print(f"  Moral: {story.moral}")
+
+
+if __name__ == "__main__":
+    # Run examples
+    # Note: These require a working OpenAI API key or local LLM server
+
+    try:
+        example_temperature_range_text()
+    except Exception as e:
+        print(f"Example 1 failed: {e}\n")
+
+    try:
+        example_temperature_range_pydantic()
+    except Exception as e:
+        print(f"Example 2 failed: {e}\n")
+
+    try:
+        example_two_step_parsing()
+    except Exception as e:
+        print(f"Example 3 failed: {e}\n")
+
+    print("\n" + "=" * 60)
+    print("Examples complete!")
+    print("=" * 60)

speedy_utils-1.1.24/examples_improved_error_tracing.py

@@ -0,0 +1,85 @@
+"""
+Direct comparison: Before and After error tracing improvements.
+
+This demonstrates the exact improvement for the user's original error case.
+"""
+
+from speedy_utils import multi_thread
+
+
+def simulate_original_error():
+    """
+    Simulates the exact error from the user's example:
+    - User has a function that creates lambda functions
+    - Accidentally passes list of functions as 'func' parameter
+    - Gets TypeError: 'list' object is not callable
+    """
+    
+    def lm_translate(msgs, temperature, max_tokens):
+        """Mock language model translate function."""
+        return [{'parsed': f'translation at temp={temperature:.2f}'}]
+    
+    def translate(n=5, max_temperature=1.0):
+        """Function that generates choices with different temperatures."""
+        step = max_temperature / n
+        fns = []
+        target_text = 'Some text to translate'
+        msgs = [{'role': 'user', 'content': 'Translate this'}]
+        
+        for i in range(n):
+            fn = lambda x: lm_translate(
+                msgs,
+                temperature=0.1 + 0.1 * i * step,
+                max_tokens=len(target_text) + 32,
+            )[0]
+            fns.append(fn)
+        
+        # THE BUG: User passed fns (a list) as the func parameter
+        # Should be: multi_thread(some_function, fns)
+        # Instead did: multi_thread(fns, range(n))
+        choices = multi_thread(fns, range(n), progress=False)
+        return choices
+    
+    return translate()
+
+
+def main():
+    print('='*70)
+    print('BEFORE vs AFTER: Error Tracing Improvements')
+    print('='*70)
+    
+    print('\nBEFORE (old behavior):')
+    print('-' * 70)
+    print('''
+The error traceback showed:
+  - Line in multi_thread.py:474
+  - concurrent.futures/_base.py:449
+  - concurrent.futures/thread.py:59
+  - multi_worker/thread.py:155
+  - ... many infrastructure frames ...
+  - Finally: TypeError: 'list' object is not callable
+
+User had to dig through 10+ lines of infrastructure code
+to find the actual problem.
+''')
+    
+    print('\nAFTER (new behavior):')
+    print('-' * 70)
+    
+    try:
+        simulate_original_error()
+    except TypeError as e:
+        print(f'\n{type(e).__name__}: {e}\n')
+    
+    print('-' * 70)
+    print('\nKey differences:')
+    print('  ✓ Immediate identification of the problem')
+    print('  ✓ Clear hint about what went wrong')
+    print('  ✓ Shows exactly what was passed (list of functions)')
+    print('  ✓ No infrastructure clutter')
+    print('  ✓ Debugging time: < 5 seconds vs > 1 minute')
+    print('='*70)
+
+
+if __name__ == '__main__':
+    main()

speedy_utils-1.1.24/notebooks/llm_utils/llm_as_a_judge.ipynb

@@ -0,0 +1,300 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "136ff273",
+   "metadata": {},
+   "source": [
+    "# LLM-as-a-Judge Tutorial\n",
+    "\n",
+    "This notebook demonstrates how to use the LLM-as-a-Judge system with structured prompts, variable substitution, and SFT export capabilities."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "60c1bf59",
+   "metadata": {},
+   "source": [
+    "## Setup and Imports"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "bb8f8e2b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "from llm_utils import (\n",
+    "    LLMJudgeBase, \n",
+    "    Signature, \n",
+    "    InputField, \n",
+    "    OutputField\n",
+    ")\n",
+    "from pydantic import BaseModel\n",
+    "import json"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eaceb8bd",
+   "metadata": {},
+   "source": [
+    "## Example 1: DSPy-like Signature System\n",
+    "\n",
+    "First, let's create a simple factual accuracy judge using the Signature system:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "5b5e2123",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Generated Instruction:\n",
+      "Judge if the answer is factually correct based on the context.\n",
+      "\n",
+      "**Input Fields:**\n",
+      "- context (str): Context for the prediction\n",
+      "- question (str): Question to be answered\n",
+      "- answer (str): Answer for the question\n",
+      "\n",
+      "**Output Fields:**\n",
+      "- factually_correct (bool): Is the answer factually correct based on the context?\n",
+      "\n",
+      "\n",
+      "==================================================\n",
+      "\n",
+      "Input Schema:\n",
+      "{\n",
+      "  \"properties\": {\n",
+      "    \"context\": {\n",
+      "      \"description\": \"Context for the prediction\",\n",
+      "      \"title\": \"Context\",\n",
+      "      \"type\": \"string\"\n",
+      "    },\n",
+      "    \"question\": {\n",
+      "      \"description\": \"Question to be answered\",\n",
+      "      \"title\": \"Question\",\n",
+      "      \"type\": \"string\"\n",
+      "    },\n",
+      "    \"answer\": {\n",
+      "      \"description\": \"Answer for the question\",\n",
+      "      \"title\": \"Answer\",\n",
+      "      \"type\": \"string\"\n",
+      "    }\n",
+      "  },\n",
+      "  \"required\": [\n",
+      "    \"context\",\n",
+      "    \"question\",\n",
+      "    \"answer\"\n",
+      "  ],\n",
+      "  \"title\": \"FactJudgeInput\",\n",
+      "  \"type\": \"object\"\n",
+      "}\n",
+      "\n",
+      "Output Schema:\n",
+      "{\n",
+      "  \"properties\": {\n",
+      "    \"factually_correct\": {\n",
+      "      \"description\": \"Is the answer factually correct based on the context?\",\n",
+      "      \"title\": \"Factually Correct\",\n",
+      "      \"type\": \"boolean\"\n",
+      "    }\n",
+      "  },\n",
+      "  \"required\": [\n",
+      "    \"factually_correct\"\n",
+      "  ],\n",
+      "  \"title\": \"FactJudgeOutput\",\n",
+      "  \"type\": \"object\"\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Define a signature like DSPy (original syntax - no more type warnings!)\n",
+    "class FactJudge(Signature):\n",
+    "    \"\"\"Judge if the answer is factually correct based on the context.\"\"\"\n",
+    "    \n",
+    "    # No more type warnings with the updated InputField/OutputField!\n",
+    "    context: str = InputField(desc=\"Context for the prediction\")\n",
+    "    question: str = InputField(desc=\"Question to be answered\")\n",
+    "    answer: str = InputField(desc=\"Answer for the question\")\n",
+    "    factually_correct: bool = OutputField(desc=\"Is the answer factually correct based on the context?\")\n",
+    "\n",
+    "# Show the generated instruction\n",
+    "print(\"Generated Instruction:\")\n",
+    "print(FactJudge.get_instruction())\n",
+    "print(\"\\n\" + \"=\"*50 + \"\\n\")\n",
+    "\n",
+    "# Show the input/output models (now always Pydantic models)\n",
+    "input_model = FactJudge.get_input_model()\n",
+    "output_model = FactJudge.get_output_model()\n",
+    "\n",
+    "print(\"Input Schema:\")\n",
+    "print(json.dumps(input_model.model_json_schema(), indent=2))\n",
+    "\n",
+    "print(\"\\nOutput Schema:\")\n",
+    "print(json.dumps(output_model.model_json_schema(), indent=2))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "380542eb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Sig(Signature):\n",
+    "    \"\"\"You are a careful **translation evaluator**.\n",
+    "\n",
+    "You are given five inputs:\n",
+    "\n",
+    "* **Source Prompt** (the original text & any constraints)\n",
+    "* **AI Translation** (the machine translation to evaluate)\n",
+    "* **Human Reference** (a reference rendering; use only for guidance, not as ground truth)\n",
+    "* **System Message** (an automated hint about a possible structural error)\n",
+    "* **Glossaries** (optional terminology constraints; may be empty)\n",
+    "\n",
+    "## Your tasks\n",
+    "\n",
+    "1. **Check structure correctness**:\n",
+    "   - Use the System Message as a hint.\n",
+    "   - Assign a `structure_score`:\n",
+    "     * `0` = structure is clearly wrong or the error flagged is correct.\n",
+    "     * `1` = partially correct but flawed.\n",
+    "     * `2` = structure is correct; the system error is invalid.\n",
+    "\n",
+    "2. **Check translation quality**:\n",
+    "   - Compare AI Translation with Source Prompt and Human Reference.\n",
+    "   - Assign a `translation_score`:\n",
+    "     * `0` = unfaithful (major omissions/additions/distortions/repetitions).\n",
+    "     * `1` = somewhat faithful (mostly correct but noticeable issues).\n",
+    "     * `2` = faithful (preserves meaning, scope, nuance; only minor style differences).\n",
+    "\n",
+    "3. **Check glossary/terminology adherence**:\n",
+    "   - If no glossary is provided → `term_score = 2`.\n",
+    "   - If glossary exists but only partially followed → `term_score = 1`.\n",
+    "   - If glossary exists but not followed at all → `term_score = 0`.\n",
+    "\n",
+    "## Output format (JSON only; no commentary)\n",
+    "\n",
+    "{{\"structure_score\": <0|1|2>, \"translation_score\": <0|1|2>, \"term_score\": <0|1|2>}}\n",
+    "\n",
+    "* Return exactly one JSON object.\n",
+    "* Do not output any explanations.\n",
+    "\"\"\"\n",
+    "    SOURCE_PROMPT: str = InputField(desc=\"The original text to be translated, along with any constraints.\")\n",
+    "    AI_TRANSLATION: str = InputField(desc=\"The machine translation output to be evaluated.\")\n",
+    "    HUMAN_REFERENCE: str = InputField(desc=\"A reference human translation, to be used for guidance but not as ground truth.\")\n",
+    "    SYSTEM_MESSAGE: str = InputField(desc=\"An automated hint about a possible structural error in the AI translation.\")\n",
+    "    GLOSSARIES: str = InputField(desc=\"Optional terminology constraints; may be empty.\")\n",
+    "    \n",
+    "    structure_score: int = OutputField(desc=\"Score for structural correctness: 0 (wrong), 1 (partially correct), 2 (correct)\")\n",
+    "    glossary_score: int = OutputField(desc=\"Score for glossary adherence: 0 (not followed), 1 (partially followed), 2 (fully followed or no glossary)\")\n",
+    "    translation_score: int = OutputField(desc=\"Score for translation quality: 0 (unfaithful), 1 (somewhat faithful), 2 (faithful)\")\n",
+    "        \n",
+    "# --- Updated evaluation prompt ---\n",
+    "\n",
+    "import os\n",
+    "judge = LLMJudgeBase(signature=Sig, client=8000) # vllm is hosted at port 8000\n",
+    "judge = LLMJudgeBase(signature=Sig, model='gpt-4.1-mini', client=None) # use openai's gpt-4.1 model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "288ebc9b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_data = Sig.get_input_model()(\n",
+    "    SOURCE_PROMPT=\"Translate the following English text to French, ensuring that the structure is preserved and the terminology is accurate. The text is: 'The quick brown fox jumps over the lazy dog.'\",\n",
+    "    AI_TRANSLATION=\"Le renard brun rapide saute par-dessus le chien paresseux.\",\n",
+    "    HUMAN_REFERENCE=\"Le vif renard brun bondit par-dessus le chien paresseux.\",\n",
+    "    SYSTEM_MESSAGE=\"The AI translation has a structural error: it uses 'rapide' instead of 'vif' to describe the fox, which affects the nuance of the sentence.\",\n",
+    "    GLOSSARIES=\"vif: quick, lively; paresseux: lazy\",\n",
+    ")\n",
+    "output = judge(input_data)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "70d221c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "ename": "AttributeError",
+     "evalue": "'LLMJudgeBase' object has no attribute 'inspect_history'",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
+      "\u001b[31mAttributeError\u001b[39m                            Traceback (most recent call last)",
+      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[28]\u001b[39m\u001b[32m, line 1\u001b[39m\n\u001b[32m----> \u001b[39m\u001b[32m1\u001b[39m \u001b[43mjudge\u001b[49m\u001b[43m.\u001b[49m\u001b[43minspect_history\u001b[49m()\n",
+      "\u001b[31mAttributeError\u001b[39m: 'LLMJudgeBase' object has no attribute 'inspect_history'"
+     ]
+    }
+   ],
+   "source": [
+    "judge.inspect_history()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "59a2c995",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "Okay, let's start by looking at the structure. The system message says the AI translation used 'rapide' instead of 'vif', which affects the nuance. The original prompt specified using accurate terminology. The glossary provides 'vif' for 'quick', so the AI should have used 'vif' instead of 'rapide'. That's a structural issue because the term choice is specified. So structure_score is 0.\n",
+      "\n",
+      "Next, translation quality. The AI's translation is mostly correct but uses the wrong term. The human reference uses 'vif', which is the correct term according to the glossary. The meaning is preserved, but the nuance is off. So translation_score is 1 because it's somewhat faithful but has a noticeable issue.\n",
+      "\n",
+      "Glossary adherence: The glossary specifies 'vif' for 'quick', but the AI used 'rapide'. So the term wasn't followed. Term_score is 0.\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2b16f277",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "speedy-utils",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

{speedy_utils-1.1.23 → speedy_utils-1.1.24}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "speedy-utils"
-version = "1.1.23"
+version = "1.1.24"
 description = "Fast and easy-to-use package for data science"
 authors = [{ name = "AnhVTH", email = "anhvth.226@gmail.com" }]
 readme = "README.md"