scorebook 0.0.14__py3-none-any.whl → 0.0.16__py3-none-any.whl

tutorials/notebooks/3.1-adaptive_evaluation_phi.ipynb

@@ -0,0 +1,354 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cell-0",
+   "metadata": {},
+   "source": [
+    "# Adaptive Evaluations with Scorebook (Phi)\n",
+    "\n",
+    "This notebook demonstrates Trismik's adaptive evaluation feature using a **local open-source model** that runs on your machine without requiring API keys or cloud costs.\n",
+    "\n",
+    "## What are Adaptive Evaluations?\n",
+    "\n",
+    "Adaptive evaluations dynamically select questions based on a model's previous responses, similar to adaptive testing in education (like the GRE or GMAT).\n",
+    "\n",
+    "### Benefits:\n",
+    "- **More efficient**: Fewer questions needed to assess capability\n",
+    "- **Precise measurement**: Better statistical confidence intervals\n",
+    "- **Optimal difficulty**: Questions adapt to the model's skill level\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "- **Trismik API key**: Get yours at https://app.trismik.com/settings\n",
+    "- **Trismik Project**: Create a project at https://app.trismik.com and copy its Project ID\n",
+    "- **Hardware**: GPU recommended but not required (CPU inference will be slower)\n",
+    "- **Packages**: `pip install transformers torch` (or `pip install transformers torch torchvision` for full PyTorch)\n",
+    "\n",
+    "## Note on Model Performance\n",
+    "\n",
+    "⚠️ **Important**: Local models (especially smaller ones) may not perform as well on complex reasoning tasks. This notebook prioritizes **accessibility and reproducibility** over maximum accuracy and uses microsoft Phi-3 on MMLU-Pro."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-1",
+   "metadata": {},
+   "source": [
+    "## Setup Credentials\n",
+    "\n",
+    "Set your Trismik credentials here:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-2",
+   "metadata": {},
+   "source": "# STEP 1: Get your Trismik API key from https://app.trismik.com/settings\n# STEP 2: Create a project at https://app.trismik.com and copy the Project ID\n\n# Set your credentials here\nTRISMIK_API_KEY = \"your-trismik-api-key\"\nTRISMIK_PROJECT_ID = \"your-project-id\"",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-3",
+   "metadata": {},
+   "source": [
+    "## Import Dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-4",
+   "metadata": {},
+   "source": [
+    "import asyncio\n",
+    "import string\n",
+    "from pprint import pprint\n",
+    "from typing import Any, List\n",
+    "\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
+    "\n",
+    "from scorebook import evaluate_async, login"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-5",
+   "metadata": {},
+   "source": [
+    "## Login to Trismik\n",
+    "\n",
+    "Authenticate with your Trismik account:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-6",
+   "metadata": {},
+   "source": [
+    "if not TRISMIK_API_KEY or TRISMIK_API_KEY == \"your-trismik-api-key\":\n",
+    "    raise ValueError(\"Please set TRISMIK_API_KEY. Get your API key from https://app.trismik.com/settings\")\n",
+    "\n",
+    "login(TRISMIK_API_KEY)\n",
+    "print(\"✓ Logged in to Trismik\")\n",
+    "\n",
+    "if not TRISMIK_PROJECT_ID or TRISMIK_PROJECT_ID == \"your-project-id\":\n",
+    "    raise ValueError(\"Please set TRISMIK_PROJECT_ID. Create a project at https://app.trismik.com\")\n",
+    "\n",
+    "print(f\"✓ Using project: {TRISMIK_PROJECT_ID}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-7",
+   "metadata": {},
+   "source": [
+    "## Initialize Local Model\n",
+    "\n",
+    "We'll use Phi-3-mini, a compact 3.8B parameter model that runs efficiently on consumer hardware.\n",
+    "\n",
+    "**Model Options** (in order of size/performance):\n",
+    "- `microsoft/Phi-3-mini-4k-instruct` (3.8B) - Fast, runs on most hardware\n",
+    "- `microsoft/Phi-3-small-8k-instruct` (7B) - Better performance, needs more memory\n",
+    "- `microsoft/Phi-3-medium-4k-instruct` (14B) - High performance, requires GPU\n",
+    "\n",
+    "Change the model name below based on your hardware capabilities."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-8",
+   "metadata": {},
+   "source": [
+    "# Select model (change based on your hardware)\n",
+    "model_name = \"microsoft/Phi-3-mini-4k-instruct\"\n",
+    "\n",
+    "print(f\"Loading {model_name}...\")\n",
+    "print(\"(This may take a few minutes on first run as the model downloads)\\n\")\n",
+    "\n",
+    "# Check if CUDA is available\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Using device: {device}\")\n",
+    "\n",
+    "# Load tokenizer and model\n",
+    "tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    model_name,\n",
+    "    torch_dtype=torch.float16 if device == \"cuda\" else torch.float32,\n",
+    "    device_map=\"auto\" if device == \"cuda\" else None,\n",
+    "    trust_remote_code=True,\n",
+    ")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    model = model.to(device)\n",
+    "\n",
+    "print(f\"\\n✓ Model loaded successfully on {device}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-9",
+   "metadata": {},
+   "source": [
+    "## Define Async Inference Function\n",
+    "\n",
+    "Create an async function to process inputs through the local model:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-10",
+   "metadata": {},
+   "source": [
+    "async def inference(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Process inputs through the local Phi model.\n",
+    "    \n",
+    "    Args:\n",
+    "        inputs: Input values from an EvalDataset. For adaptive MMLU-Pro,\n",
+    "               each input is a dict with 'question' and 'options' keys.\n",
+    "        hyperparameters: Model hyperparameters.\n",
+    "    \n",
+    "    Returns:\n",
+    "        List of model outputs for all inputs.\n",
+    "    \"\"\"\n",
+    "    outputs = []\n",
+    "    \n",
+    "    for input_val in inputs:\n",
+    "        # Handle dict input from adaptive dataset\n",
+    "        if isinstance(input_val, dict):\n",
+    "            prompt = input_val.get(\"question\", \"\")\n",
+    "            if \"options\" in input_val:\n",
+    "                prompt += \"\\nOptions:\\n\" + \"\\n\".join(\n",
+    "                    f\"{letter}: {choice}\"\n",
+    "                    for letter, choice in zip(string.ascii_uppercase, input_val[\"options\"])\n",
+    "                )\n",
+    "        else:\n",
+    "            prompt = str(input_val)\n",
+    "        \n",
+    "        # Build prompt for Phi model\n",
+    "        system_message = \"Answer the question with a single letter representing the correct answer from the list of choices. Do not provide any additional explanation or output beyond the single letter.\"\n",
+    "        \n",
+    "        # Phi-3 uses ChatML format\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": system_message},\n",
+    "            {\"role\": \"user\", \"content\": prompt},\n",
+    "        ]\n",
+    "        \n",
+    "        # Apply chat template\n",
+    "        formatted_prompt = tokenizer.apply_chat_template(\n",
+    "            messages, tokenize=False, add_generation_prompt=True\n",
+    "        )\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        inputs_tokenized = tokenizer(\n",
+    "            formatted_prompt, return_tensors=\"pt\", truncation=True, max_length=2048\n",
+    "        )\n",
+    "        inputs_tokenized = {k: v.to(device) for k, v in inputs_tokenized.items()}\n",
+    "        \n",
+    "        # Generate\n",
+    "        try:\n",
+    "            with torch.no_grad():\n",
+    "                output_ids = model.generate(\n",
+    "                    **inputs_tokenized,\n",
+    "                    max_new_tokens=10,  # We only need 1 letter\n",
+    "                    temperature=0.7,\n",
+    "                    do_sample=True,\n",
+    "                    pad_token_id=tokenizer.eos_token_id,\n",
+    "                )\n",
+    "            \n",
+    "            # Decode only the generated tokens\n",
+    "            generated_tokens = output_ids[0][inputs_tokenized[\"input_ids\"].shape[1]:]\n",
+    "            output = tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()\n",
+    "            \n",
+    "            # Extract just the first letter if model outputs more\n",
+    "            if output and output[0].upper() in string.ascii_uppercase:\n",
+    "                output = output[0].upper()\n",
+    "        except Exception as e:\n",
+    "            output = f\"Error: {str(e)}\"\n",
+    "        \n",
+    "        outputs.append(output)\n",
+    "    \n",
+    "    return outputs"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-11",
+   "metadata": {},
+   "source": [
+    "## Run Adaptive Evaluation\n",
+    "\n",
+    "Use `evaluate_async()` with an adaptive dataset (indicated by the `:adaptive` suffix):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-12",
+   "metadata": {},
+   "source": [
+    "print(f\"Running adaptive evaluation on Common Sense QA with model: {model_name.split('/')[-1]}\")\n",
+    "print(\"Note: Adaptive evaluation selects questions dynamically based on responses.\")\n",
+    "\n",
+    "# Run adaptive evaluation\n",
+    "results = await evaluate_async(\n",
+    "    inference,\n",
+    "    datasets=\"trismik/CommonSenseQA:adaptive\",  # Adaptive datasets have the \":adaptive\" suffix\n",
+    "    experiment_id=\"Adaptive-Common-Sense-QA-Local-Notebook\",\n",
+    "    project_id=TRISMIK_PROJECT_ID,\n",
+    "    return_dict=True,\n",
+    "    return_aggregates=True,\n",
+    "    return_items=True,\n",
+    "    return_output=True,\n",
+    ")\n",
+    "\n",
+    "print(\"\\n✓ Adaptive evaluation complete!\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-13",
+   "metadata": {},
+   "source": [
+    "## View Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "cell-14",
+   "metadata": {},
+   "source": [
+    "pprint(results)"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-17",
+   "metadata": {},
+   "source": [
+    "## View Results on Dashboard\n",
+    "\n",
+    "Your results have been uploaded to Trismik's dashboard for visualization and tracking:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-19",
+   "metadata": {},
+   "source": [
+    "## Understanding Adaptive Testing\n",
+    "\n",
+    "### How it works:\n",
+    "1. **Initial Questions**: Start with medium-difficulty questions\n",
+    "2. **Adaptation**: If the model answers correctly, harder questions follow; if incorrect, easier questions are selected\n",
+    "3. **Convergence**: The test converges to the model's true ability level\n",
+    "4. **Stopping Criteria**: Stops when sufficient confidence is reached\n",
+    "\n",
+    "### Benefits vs. Traditional Testing:\n",
+    "- **Efficiency**: Typically requires 50-70% fewer questions for the same precision\n",
+    "- **Precision**: Better estimates of model capability\n",
+    "- **Engagement**: Questions are appropriately challenging\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "- Try different local models (Phi-3-small, Phi-3-medium, Llama-3, etc.)\n",
+    "- Compare local model performance with the OpenAI version\n",
+    "- Explore other adaptive datasets available on Trismik\n",
+    "- See the **Upload Results** notebook for non-adaptive result tracking"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

tutorials/notebooks/3.2-adaptive_evaluation_gpt.ipynb

@@ -0,0 +1,243 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Adaptive Evaluations with Scorebook (GPT)\n",
+    "\n",
+    "This notebook demonstrates Trismik's adaptive evaluation feature using **OpenAI's GPT models** for high-accuracy results.\n",
+    "\n",
+    "> **Looking for a version without API costs?** See `3-adaptive_evaluation_local.ipynb` for a version using local open-source models (Phi-3) that runs on your machine without API keys.\n",
+    "\n",
+    "## What are Adaptive Evaluations?\n",
+    "\n",
+    "Adaptive evaluations dynamically select questions based on a model's previous responses, similar to adaptive testing in education (like the GRE or GMAT).\n",
+    "\n",
+    "### Benefits:\n",
+    "- **More efficient**: Fewer questions needed to assess capability\n",
+    "- **Precise measurement**: Better statistical confidence intervals\n",
+    "- **Optimal difficulty**: Questions adapt to the model's skill level\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "- **Trismik API key**: Get yours at https://app.trismik.com/settings\n",
+    "- **Trismik Project**: Create a project at https://app.trismik.com and copy its Project ID\n",
+    "- **OpenAI API key**: For high-accuracy results on complex reasoning tasks"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Setup Credentials\n\nSet your API credentials here:"
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "# STEP 1: Get your Trismik API key from https://app.trismik.com/settings\n# STEP 2: Create a project at https://app.trismik.com and copy the Project ID\n# STEP 3: Get your OpenAI API key from https://platform.openai.com/api-keys\n\n# Set your credentials here\nTRISMIK_API_KEY = \"your-trismik-api-key\"\nTRISMIK_PROJECT_ID = \"your-project-id\"\nOPENAI_API_KEY = \"your-openai-api-key\"",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Import Dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "import asyncio\n",
+    "import string\n",
+    "from pprint import pprint\n",
+    "from typing import Any, List\n",
+    "\n",
+    "from openai import AsyncOpenAI\n",
+    "from scorebook import evaluate_async, login"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Login to Trismik\n",
+    "\n",
+    "Authenticate with your Trismik account:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "if not TRISMIK_API_KEY or TRISMIK_API_KEY == \"your-trismik-api-key\":\n    raise ValueError(\"Please set TRISMIK_API_KEY. Get your API key from https://app.trismik.com/settings\")\n\nlogin(TRISMIK_API_KEY)\nprint(\"✓ Logged in to Trismik\")\n\nif not TRISMIK_PROJECT_ID or TRISMIK_PROJECT_ID == \"your-project-id\":\n    raise ValueError(\"Please set TRISMIK_PROJECT_ID. Create a project at https://app.trismik.com\")\n\nprint(f\"✓ Using project: {TRISMIK_PROJECT_ID}\")",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Initialize OpenAI Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "if not OPENAI_API_KEY or OPENAI_API_KEY == \"your-openai-api-key\":\n    raise ValueError(\"Please set OPENAI_API_KEY. Get your API key from https://platform.openai.com/api-keys\")\n\nclient = AsyncOpenAI(api_key=OPENAI_API_KEY)  # pragma: allowlist secret\nmodel_name = \"gpt-4o-mini\"\n\nprint(f\"✓ Using model: {model_name}\")",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define Async Inference Function\n",
+    "\n",
+    "Create an async function to process inputs through the OpenAI API:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "async def inference(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Process inputs through OpenAI's API.\n",
+    "    \n",
+    "    Args:\n",
+    "        inputs: Input values from an EvalDataset. For adaptive MMLU-Pro,\n",
+    "               each input is a dict with 'question' and 'options' keys.\n",
+    "        hyperparameters: Model hyperparameters.\n",
+    "    \n",
+    "    Returns:\n",
+    "        List of model outputs for all inputs.\n",
+    "    \"\"\"\n",
+    "    outputs = []\n",
+    "    \n",
+    "    for input_val in inputs:\n",
+    "        # Handle dict input from adaptive dataset\n",
+    "        if isinstance(input_val, dict):\n",
+    "            prompt = input_val.get(\"question\", \"\")\n",
+    "            if \"options\" in input_val:\n",
+    "                prompt += \"\\nOptions:\\n\" + \"\\n\".join(\n",
+    "                    f\"{letter}: {choice}\"\n",
+    "                    for letter, choice in zip(string.ascii_uppercase, input_val[\"options\"])\n",
+    "                )\n",
+    "        else:\n",
+    "            prompt = str(input_val)\n",
+    "        \n",
+    "        # Build messages for OpenAI API\n",
+    "        messages = [\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": \"Answer the question with a single letter representing the correct answer from the list of choices. Do not provide any additional explanation or output beyond the single letter.\",\n",
+    "            },\n",
+    "            {\"role\": \"user\", \"content\": prompt},\n",
+    "        ]\n",
+    "        \n",
+    "        # Call OpenAI API\n",
+    "        try:\n",
+    "            response = await client.chat.completions.create(\n",
+    "                model=model_name,\n",
+    "                messages=messages,\n",
+    "                temperature=0.7,\n",
+    "            )\n",
+    "            output = response.choices[0].message.content.strip()\n",
+    "        except Exception as e:\n",
+    "            output = f\"Error: {str(e)}\"\n",
+    "        \n",
+    "        outputs.append(output)\n",
+    "    \n",
+    "    return outputs"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Run Adaptive Evaluation\n",
+    "\n",
+    "Use `evaluate_async()` with an adaptive dataset (indicated by the `:adaptive` suffix):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "print(f\"Running adaptive evaluation on Common Sense QA with model: {model_name}\")\n",
+    "print(\"Note: Adaptive evaluation selects questions dynamically based on responses.\\n\")\n",
+    "\n",
+    "# Run adaptive evaluation\n",
+    "results = await evaluate_async(\n",
+    "    inference,\n",
+    "    datasets=\"trismik/CommonSenseQA:adaptive\",  # Adaptive datasets have the \":adaptive\" suffix\n",
+    "    experiment_id=\"Adaptive-Common-Sense-QA-Notebook\",\n",
+    "    project_id=TRISMIK_PROJECT_ID,\n",
+    "    return_dict=True,\n",
+    "    return_aggregates=True,\n",
+    "    return_items=True,\n",
+    "    return_output=True,\n",
+    ")\n",
+    "\n",
+    "print(\"\\n✓ Adaptive evaluation complete!\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## View Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "pprint(results)"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## View Results on Dashboard\n",
+    "\n",
+    "Your results have been uploaded to Trismik's dashboard for visualization and tracking:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Understanding Adaptive Testing\n\n### How it works:\n1. **Initial Questions**: Start with medium-difficulty questions\n2. **Adaptation**: If the model answers correctly, harder questions follow; if incorrect, easier questions are selected\n3. **Convergence**: The test converges to the model's true ability level\n4. **Stopping Criteria**: Stops when sufficient confidence is reached\n\n### Benefits vs. Traditional Testing:\n- **Efficiency**: Typically requires 50-70% fewer questions for the same precision\n- **Precision**: Better estimates of model capability\n- **Engagement**: Questions are appropriately challenging\n\n## Next Steps\n\n- Try adaptive evaluation with different models to compare\n- **Don't have an OpenAI API key?** See `3-adaptive_evaluation_local.ipynb` to run adaptive evaluations with local open-source models (Phi-3, Llama, etc.)\n- Explore other adaptive datasets available on Trismik\n- See the **Upload Results** notebook for non-adaptive result tracking"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}