scorebook 0.0.13__py3-none-any.whl → 0.0.15__py3-none-any.whl

tutorials/notebooks/2-evaluating.ipynb

@@ -0,0 +1,316 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Evaluating Models with Scorebook\n",
+    "\n",
+    "This notebook demonstrates how to use Scorebook's `evaluate()` function to run inference and compute metrics in a single step.\n",
+    "\n",
+    "## When to use `evaluate()`\n",
+    "\n",
+    "- You want to run inference on a dataset and score the results\n",
+    "- You're comparing different models on the same dataset\n",
+    "- You want to track hyperparameters alongside results\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "This example uses a local HuggingFace model. For cloud models (OpenAI), see the examples directory."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "Import necessary modules:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "from pprint import pprint\n",
+    "from typing import Any, List\n",
+    "import transformers\n",
+    "\n",
+    "from scorebook import EvalDataset, evaluate"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Initialize Your Model\n",
+    "\n",
+    "Set up a HuggingFace pipeline for inference:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "model_name = \"microsoft/Phi-4-mini-instruct\"\n",
+    "\n",
+    "pipeline = transformers.pipeline(\n",
+    "    \"text-generation\",\n",
+    "    model=model_name,\n",
+    "    model_kwargs={\"torch_dtype\": \"auto\"},\n",
+    "    device_map=\"auto\",\n",
+    ")\n",
+    "\n",
+    "print(f\"Model loaded: {model_name}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define Your Inference Function\n",
+    "\n",
+    "Create a function that processes inputs and returns outputs:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "def inference(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Process inputs through the model.\n",
+    "    \n",
+    "    Args:\n",
+    "        inputs: List of input values from the dataset\n",
+    "        hyperparameters: Model hyperparameters (e.g., temperature, system_message)\n",
+    "        \n",
+    "    Returns:\n",
+    "        List of model outputs\n",
+    "    \"\"\"\n",
+    "    outputs = []\n",
+    "    \n",
+    "    for input_val in inputs:\n",
+    "        # Build messages for the model\n",
+    "        messages = [\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": hyperparameters.get(\"system_message\", \"You are a helpful assistant.\")\n",
+    "            },\n",
+    "            {\"role\": \"user\", \"content\": str(input_val)},\n",
+    "        ]\n",
+    "        \n",
+    "        # Run inference\n",
+    "        result = pipeline(\n",
+    "            messages,\n",
+    "            max_new_tokens=hyperparameters.get(\"max_new_tokens\", 100),\n",
+    "        )\n",
+    "        \n",
+    "        # Extract the answer\n",
+    "        output = str(result[0][\"generated_text\"][-1][\"content\"])\n",
+    "        outputs.append(output)\n",
+    "    \n",
+    "    return outputs"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Load Your Dataset\n",
+    "\n",
+    "Create an evaluation dataset from a JSON file:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "# Create a sample dataset\n",
+    "sample_data = [\n",
+    "    {\"question\": \"What is 2 + 2?\", \"answer\": \"4\"},\n",
+    "    {\"question\": \"What is the capital of France?\", \"answer\": \"Paris\"},\n",
+    "    {\"question\": \"Who wrote Romeo and Juliet?\", \"answer\": \"William Shakespeare\"},\n",
+    "]\n",
+    "\n",
+    "# Create EvalDataset directly from list\n",
+    "dataset = EvalDataset.from_list(\n",
+    "    name=\"sample_questions\",\n",
+    "    metrics=\"accuracy\",\n",
+    "    items=sample_data,\n",
+    "    input=\"question\",\n",
+    "    label=\"answer\",\n",
+    ")\n",
+    "\n",
+    "print(f\"Loaded dataset with {len(dataset.items)} items\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Run Evaluation\n",
+    "\n",
+    "Use `evaluate()` to run inference and compute metrics:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "results = evaluate(\n",
+    "    inference,\n",
+    "    dataset,\n",
+    "    hyperparameters={\n",
+    "        \"system_message\": \"Answer the question directly and concisely.\",\n",
+    "        \"max_new_tokens\": 50,\n",
+    "    },\n",
+    "    return_aggregates=True,\n",
+    "    return_items=True,\n",
+    "    return_output=True,\n",
+    "    upload_results=False,  # Set to True to upload to Trismik\n",
+    ")\n",
+    "\n",
+    "pprint(results)"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Analyze Results\n",
+    "\n",
+    "Examine the outputs and metrics:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "# Overall accuracy\n",
+    "print(f\"\\nOverall Accuracy: {results['aggregate_results'][0]['accuracy']:.2%}\")\n",
+    "\n",
+    "# Per-item results\n",
+    "print(\"\\nPer-Item Results:\")\n",
+    "for i, item in enumerate(results['item_results'], 1):\n",
+    "    print(f\"\\nQuestion {i}: {item['input']}\")\n",
+    "    print(f\"  Model Output: {item['output']}\")\n",
+    "    print(f\"  Expected: {item['label']}\")\n",
+    "    print(f\"  Correct: {'✓' if item['accuracy'] == 1.0 else '✗'}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "## Hyperparameter Sweeps\n",
+    "\n",
+    "Evaluate with different hyperparameters to find optimal settings:"
+   ]
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "source": [
+    "# Define an inference function\n",
+    "def inference(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Process inputs through the model.\n",
+    "\n",
+    "    Args:\n",
+    "        inputs: Input values from an EvalDataset.\n",
+    "        hyperparameters: Model hyperparameters including system_message, temperature, top_p, top_k.\n",
+    "\n",
+    "    Returns:\n",
+    "        List of model outputs for all inputs.\n",
+    "    \"\"\"\n",
+    "    outputs = []\n",
+    "    for input_val in inputs:\n",
+    "        # Preprocess: Build messages\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": hyperparameters[\"system_message\"]},\n",
+    "            {\"role\": \"user\", \"content\": str(input_val)},\n",
+    "        ]\n",
+    "\n",
+    "        # Run inference\n",
+    "        result = pipeline(\n",
+    "            messages,\n",
+    "            temperature=hyperparameters[\"temperature\"],\n",
+    "            top_p=hyperparameters.get(\"top_p\"),\n",
+    "            top_k=hyperparameters.get(\"top_k\"),\n",
+    "        )\n",
+    "\n",
+    "        # Postprocess: Extract the answer\n",
+    "        output = str(result[0][\"generated_text\"][-1][\"content\"])\n",
+    "        outputs.append(output)\n",
+    "\n",
+    "    return outputs\n",
+    "\n",
+    "# Define hyperparameters with lists of values to create a sweep\n",
+    "hyperparameters = {\n",
+    "    \"system_message\": \"Answer the question directly and concisely.\",\n",
+    "    \"temperature\": [0.6, 0.7, 0.8],\n",
+    "    \"top_p\": [0.7, 0.8, 0.9],\n",
+    "    \"top_k\": [10, 20, 30],\n",
+    "}\n",
+    "\n",
+    "# Run evaluation across all hyperparameter combinations\n",
+    "results = evaluate(\n",
+    "    inference,\n",
+    "    dataset,\n",
+    "    hyperparameters=hyperparameters,\n",
+    "    return_aggregates=True,\n",
+    "    return_items=True,\n",
+    "    return_output=True,\n",
+    "    upload_results=False,\n",
+    ")\n",
+    "\n",
+    "pprint(results)"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next Steps\n",
+    "\n",
+    "- Try the **Adaptive Evaluations** notebook for efficient testing with fewer questions\n",
+    "- See the **Upload Results** notebook to track results in Trismik's dashboard\n",
+    "- Explore batch processing for faster evaluation of large datasets"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

tutorials/notebooks/3.1-adaptive_evaluation_phi.ipynb

@@ -0,0 +1,354 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cell-0",
+   "metadata": {},
+   "source": [
+    "# Adaptive Evaluations with Scorebook (Phi)\n",
+    "\n",
+    "This notebook demonstrates Trismik's adaptive evaluation feature using a **local open-source model** that runs on your machine without requiring API keys or cloud costs.\n",
+    "\n",
+    "## What are Adaptive Evaluations?\n",
+    "\n",
+    "Adaptive evaluations dynamically select questions based on a model's previous responses, similar to adaptive testing in education (like the GRE or GMAT).\n",
+    "\n",
+    "### Benefits:\n",
+    "- **More efficient**: Fewer questions needed to assess capability\n",
+    "- **Precise measurement**: Better statistical confidence intervals\n",
+    "- **Optimal difficulty**: Questions adapt to the model's skill level\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "- **Trismik API key**: Get yours at https://app.trismik.com/settings\n",
+    "- **Trismik Project**: Create a project at https://app.trismik.com and copy its Project ID\n",
+    "- **Hardware**: GPU recommended but not required (CPU inference will be slower)\n",
+    "- **Packages**: `pip install transformers torch` (or `pip install transformers torch torchvision` for full PyTorch)\n",
+    "\n",
+    "## Note on Model Performance\n",
+    "\n",
+    "⚠️ **Important**: Local models (especially smaller ones) may not perform as well on complex reasoning tasks. This notebook prioritizes **accessibility and reproducibility** over maximum accuracy and uses microsoft Phi-3 on MMLU-Pro."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-1",
+   "metadata": {},
+   "source": [
+    "## Setup Credentials\n",
+    "\n",
+    "Set your Trismik credentials here:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-2",
+   "metadata": {},
+   "source": "# STEP 1: Get your Trismik API key from https://app.trismik.com/settings\n# STEP 2: Create a project at https://app.trismik.com and copy the Project ID\n\n# Set your credentials here\nTRISMIK_API_KEY = \"your-trismik-api-key\"\nTRISMIK_PROJECT_ID = \"your-project-id\"",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-3",
+   "metadata": {},
+   "source": [
+    "## Import Dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-4",
+   "metadata": {},
+   "source": [
+    "import asyncio\n",
+    "import string\n",
+    "from pprint import pprint\n",
+    "from typing import Any, List\n",
+    "\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
+    "\n",
+    "from scorebook import evaluate_async, login"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-5",
+   "metadata": {},
+   "source": [
+    "## Login to Trismik\n",
+    "\n",
+    "Authenticate with your Trismik account:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-6",
+   "metadata": {},
+   "source": [
+    "if not TRISMIK_API_KEY or TRISMIK_API_KEY == \"your-trismik-api-key\":\n",
+    "    raise ValueError(\"Please set TRISMIK_API_KEY. Get your API key from https://app.trismik.com/settings\")\n",
+    "\n",
+    "login(TRISMIK_API_KEY)\n",
+    "print(\"✓ Logged in to Trismik\")\n",
+    "\n",
+    "if not TRISMIK_PROJECT_ID or TRISMIK_PROJECT_ID == \"your-project-id\":\n",
+    "    raise ValueError(\"Please set TRISMIK_PROJECT_ID. Create a project at https://app.trismik.com\")\n",
+    "\n",
+    "print(f\"✓ Using project: {TRISMIK_PROJECT_ID}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-7",
+   "metadata": {},
+   "source": [
+    "## Initialize Local Model\n",
+    "\n",
+    "We'll use Phi-3-mini, a compact 3.8B parameter model that runs efficiently on consumer hardware.\n",
+    "\n",
+    "**Model Options** (in order of size/performance):\n",
+    "- `microsoft/Phi-3-mini-4k-instruct` (3.8B) - Fast, runs on most hardware\n",
+    "- `microsoft/Phi-3-small-8k-instruct` (7B) - Better performance, needs more memory\n",
+    "- `microsoft/Phi-3-medium-4k-instruct` (14B) - High performance, requires GPU\n",
+    "\n",
+    "Change the model name below based on your hardware capabilities."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-8",
+   "metadata": {},
+   "source": [
+    "# Select model (change based on your hardware)\n",
+    "model_name = \"microsoft/Phi-3-mini-4k-instruct\"\n",
+    "\n",
+    "print(f\"Loading {model_name}...\")\n",
+    "print(\"(This may take a few minutes on first run as the model downloads)\\n\")\n",
+    "\n",
+    "# Check if CUDA is available\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Using device: {device}\")\n",
+    "\n",
+    "# Load tokenizer and model\n",
+    "tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    model_name,\n",
+    "    torch_dtype=torch.float16 if device == \"cuda\" else torch.float32,\n",
+    "    device_map=\"auto\" if device == \"cuda\" else None,\n",
+    "    trust_remote_code=True,\n",
+    ")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    model = model.to(device)\n",
+    "\n",
+    "print(f\"\\n✓ Model loaded successfully on {device}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-9",
+   "metadata": {},
+   "source": [
+    "## Define Async Inference Function\n",
+    "\n",
+    "Create an async function to process inputs through the local model:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-10",
+   "metadata": {},
+   "source": [
+    "async def inference(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Process inputs through the local Phi model.\n",
+    "    \n",
+    "    Args:\n",
+    "        inputs: Input values from an EvalDataset. For adaptive MMLU-Pro,\n",
+    "               each input is a dict with 'question' and 'options' keys.\n",
+    "        hyperparameters: Model hyperparameters.\n",
+    "    \n",
+    "    Returns:\n",
+    "        List of model outputs for all inputs.\n",
+    "    \"\"\"\n",
+    "    outputs = []\n",
+    "    \n",
+    "    for input_val in inputs:\n",
+    "        # Handle dict input from adaptive dataset\n",
+    "        if isinstance(input_val, dict):\n",
+    "            prompt = input_val.get(\"question\", \"\")\n",
+    "            if \"options\" in input_val:\n",
+    "                prompt += \"\\nOptions:\\n\" + \"\\n\".join(\n",
+    "                    f\"{letter}: {choice}\"\n",
+    "                    for letter, choice in zip(string.ascii_uppercase, input_val[\"options\"])\n",
+    "                )\n",
+    "        else:\n",
+    "            prompt = str(input_val)\n",
+    "        \n",
+    "        # Build prompt for Phi model\n",
+    "        system_message = \"Answer the question with a single letter representing the correct answer from the list of choices. Do not provide any additional explanation or output beyond the single letter.\"\n",
+    "        \n",
+    "        # Phi-3 uses ChatML format\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": system_message},\n",
+    "            {\"role\": \"user\", \"content\": prompt},\n",
+    "        ]\n",
+    "        \n",
+    "        # Apply chat template\n",
+    "        formatted_prompt = tokenizer.apply_chat_template(\n",
+    "            messages, tokenize=False, add_generation_prompt=True\n",
+    "        )\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        inputs_tokenized = tokenizer(\n",
+    "            formatted_prompt, return_tensors=\"pt\", truncation=True, max_length=2048\n",
+    "        )\n",
+    "        inputs_tokenized = {k: v.to(device) for k, v in inputs_tokenized.items()}\n",
+    "        \n",
+    "        # Generate\n",
+    "        try:\n",
+    "            with torch.no_grad():\n",
+    "                output_ids = model.generate(\n",
+    "                    **inputs_tokenized,\n",
+    "                    max_new_tokens=10,  # We only need 1 letter\n",
+    "                    temperature=0.7,\n",
+    "                    do_sample=True,\n",
+    "                    pad_token_id=tokenizer.eos_token_id,\n",
+    "                )\n",
+    "            \n",
+    "            # Decode only the generated tokens\n",
+    "            generated_tokens = output_ids[0][inputs_tokenized[\"input_ids\"].shape[1]:]\n",
+    "            output = tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()\n",
+    "            \n",
+    "            # Extract just the first letter if model outputs more\n",
+    "            if output and output[0].upper() in string.ascii_uppercase:\n",
+    "                output = output[0].upper()\n",
+    "        except Exception as e:\n",
+    "            output = f\"Error: {str(e)}\"\n",
+    "        \n",
+    "        outputs.append(output)\n",
+    "    \n",
+    "    return outputs"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-11",
+   "metadata": {},
+   "source": [
+    "## Run Adaptive Evaluation\n",
+    "\n",
+    "Use `evaluate_async()` with an adaptive dataset (indicated by the `:adaptive` suffix):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-12",
+   "metadata": {},
+   "source": [
+    "print(f\"Running adaptive evaluation on Common Sense QA with model: {model_name.split('/')[-1]}\")\n",
+    "print(\"Note: Adaptive evaluation selects questions dynamically based on responses.\")\n",
+    "\n",
+    "# Run adaptive evaluation\n",
+    "results = await evaluate_async(\n",
+    "    inference,\n",
+    "    datasets=\"trismik/CommonSenseQA:adaptive\",  # Adaptive datasets have the \":adaptive\" suffix\n",
+    "    experiment_id=\"Adaptive-Common-Sense-QA-Local-Notebook\",\n",
+    "    project_id=TRISMIK_PROJECT_ID,\n",
+    "    return_dict=True,\n",
+    "    return_aggregates=True,\n",
+    "    return_items=True,\n",
+    "    return_output=True,\n",
+    ")\n",
+    "\n",
+    "print(\"\\n✓ Adaptive evaluation complete!\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-13",
+   "metadata": {},
+   "source": [
+    "## View Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-14",
+   "metadata": {},
+   "source": [
+    "pprint(results)"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-17",
+   "metadata": {},
+   "source": [
+    "## View Results on Dashboard\n",
+    "\n",
+    "Your results have been uploaded to Trismik's dashboard for visualization and tracking:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-19",
+   "metadata": {},
+   "source": [
+    "## Understanding Adaptive Testing\n",
+    "\n",
+    "### How it works:\n",
+    "1. **Initial Questions**: Start with medium-difficulty questions\n",
+    "2. **Adaptation**: If the model answers correctly, harder questions follow; if incorrect, easier questions are selected\n",
+    "3. **Convergence**: The test converges to the model's true ability level\n",
+    "4. **Stopping Criteria**: Stops when sufficient confidence is reached\n",
+    "\n",
+    "### Benefits vs. Traditional Testing:\n",
+    "- **Efficiency**: Typically requires 50-70% fewer questions for the same precision\n",
+    "- **Precision**: Better estimates of model capability\n",
+    "- **Engagement**: Questions are appropriately challenging\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "- Try different local models (Phi-3-small, Phi-3-medium, Llama-3, etc.)\n",
+    "- Compare local model performance with the OpenAI version\n",
+    "- Explore other adaptive datasets available on Trismik\n",
+    "- See the **Upload Results** notebook for non-adaptive result tracking"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}