scorebook 0.0.14__py3-none-any.whl → 0.0.15__py3-none-any.whl

tutorials/quickstarts/classical_evaluations/classical_evaluation_demo.ipynb

@@ -0,0 +1,277 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "b1ef68792a1c82f2",
+   "metadata": {},
+   "source": [
+    "# _Classical_ Evaluations with Scorebook\n",
+    "\n",
+    "Scorebook, developed by Trismik, is an open-source Python library for model evaluation. It supports both Trismik’s adaptive testing and traditional classical evaluations. In a classical evaluation, a model runs inference on every item in a dataset, and the results are scored using Scorebook’s built-in metrics, such as accuracy, to produce evaluation results. Evaluation results can be automatically uploaded to the Scorebook dashboard, organized by project, for storing, managing, and visualizing model evaluation experiments.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "- **Trismik API key**: Generate a Trismik API key from the [Trismik dashboard's settings page](https://app.trismik.com/settings).\n",
+    "- **Trismik Project Id**: We recommend you use the project id generated in the [Getting Started Quick-Start Guide](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/getting_started.ipynb).\n",
+    "\n",
+    "### Install Scorebook\n"
+   ]
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "source": "!pip install scorebook",
+   "id": "c2fecbb3d7b699b3",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "\n",
+    "\n",
+    "### Setup Credentials\n",
+    "\n",
+    "Enter your Trismik API key and project id below."
+   ],
+   "id": "8e18d0e628fdb944"
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "source": [
+    "# Set your credentials here\n",
+    "TRISMIK_API_KEY = \"your-trismik-api-key-here\"\n",
+    "TRISMIK_PROJECT_ID = \"your-trismik-project-id-key-here\""
+   ],
+   "id": "4cb833e7b092ae4",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "### Login with Trismik API Key",
+   "id": "ed4e8281ecb99685"
+  },
+  {
+   "cell_type": "code",
+   "id": "f59fb493a575d361",
+   "metadata": {},
+   "source": [
+    "from scorebook import login\n",
+    "\n",
+    "login(TRISMIK_API_KEY)\n",
+    "print(\"✓ Logged in to Trismik\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cb58f57296229115",
+   "metadata": {},
+   "source": [
+    "\n",
+    "\n",
+    "## Evaluation Datasets\n",
+    "\n",
+    "A scorebook evaluation requires an evaluation dataset, represented by the `EvalDataset` class. Evaluation datasets can be constructed via a number of factory methods. In this example we will create a basic evaluation dataset from a list of evaluation items."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "fa7c936b75c83cad",
+   "metadata": {},
+   "source": [
+    "from scorebook import EvalDataset\n",
+    "from scorebook.metrics.accuracy import Accuracy\n",
+    "\n",
+    "# Create a sample dataset from a list of multiple-choice questions\n",
+    "evaluation_items = [\n",
+    "    {\"question\": \"What is 2 + 2?\", \"answer\": \"4\"},\n",
+    "    {\"question\": \"What is the capital of France?\", \"answer\": \"Paris\"},\n",
+    "    {\"question\": \"Who wrote Romeo and Juliet?\", \"answer\": \"William Shakespeare\"},\n",
+    "    {\"question\": \"What is the chemical symbol for gold?\", \"answer\": \"Au\"}\n",
+    "]\n",
+    "\n",
+    "# Create an EvalDataset from the list\n",
+    "dataset = EvalDataset.from_list(\n",
+    "    name = \"sample_multiple_choice\",\n",
+    "    metrics = Accuracy,\n",
+    "    items = evaluation_items,\n",
+    "    input = \"question\",\n",
+    "    label = \"answer\",\n",
+    ")\n",
+    "\n",
+    "print(f\"✓ Created dataset with {len(dataset.items)} items\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "609a95a43d8cfc2c",
+   "metadata": {},
+   "source": [
+    "## Preparing Models for Evaluation\n",
+    "\n",
+    "To evaluate a model with Scorebook, it must be encapsulated within an inference function. An inference function must accept a list of model inputs, pass these to the model for inference, collect and return outputs generated.\n",
+    "\n",
+    "### Instantiate a Local Qwen Model\n",
+    "\n",
+    "For this quick-start guide, we will use the lightweight Qwen2.5 0.5B instruct model, via Hugging Face's transformers package."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "d1da8af72ef8de6f",
+   "metadata": {},
+   "source": [
+    "import transformers\n",
+    "\n",
+    "# Instantiate a model\n",
+    "pipeline = transformers.pipeline(\n",
+    "    \"text-generation\",\n",
+    "    model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n",
+    "    model_kwargs={\"torch_dtype\": \"auto\"},\n",
+    "    device_map=\"auto\",\n",
+    ")\n",
+    "\n",
+    "print(\"✓ Transformers pipeline instantiated\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3b56a72374920220",
+   "metadata": {},
+   "source": [
+    "### Define an Inference Function\n",
+    "\n",
+    "An inference function can be defined to encapsulate any model, local or cloud-hosted. There is flexibility in how an inference function can be defined, the only requirements are the function signature. An inference function must,\n",
+    "\n",
+    "Accept:\n",
+    "\n",
+    "- A list of model inputs.\n",
+    "- Hyperparameters which can be optionally accessed via kwargs.\n",
+    "\n",
+    "Return\n",
+    "\n",
+    "- A list of parsed model outputs for scoring.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "55f6d1eee2fa886e",
+   "metadata": {},
+   "source": [
+    "from typing import Any, List\n",
+    "\n",
+    "# Define an inference function for the Qwen model.\n",
+    "def qwen(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Run inference on a list of inputs using the 0.5B Qwen model.\"\"\"\n",
+    "    inference_outputs = []\n",
+    "\n",
+    "    for model_input in inputs:\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": hyperparameters.get(\"system_message\", \"You are a helpful assistant.\")},\n",
+    "            {\"role\": \"user\", \"content\": str(model_input)},\n",
+    "        ]\n",
+    "\n",
+    "        output = pipeline(\n",
+    "            messages,\n",
+    "            temperature = hyperparameters.get(\"temperature\", 0.7),\n",
+    "            top_p = hyperparameters.get(\"top_p\", 0.9),\n",
+    "            top_k = hyperparameters.get(\"top_k\", 50),\n",
+    "            max_new_tokens = 512,\n",
+    "            do_sample = hyperparameters.get(\"temperature\", 0.7) > 0,\n",
+    "        )\n",
+    "\n",
+    "        inference_outputs.append(output[0][\"generated_text\"][-1][\"content\"])\n",
+    "\n",
+    "    return inference_outputs\n",
+    "\n",
+    "print(\"✓ Inference function for Qwen2.5 0.5B defined\")\n",
+    "print(qwen([\"Hello!\"]))"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1230b9ad762482af",
+   "metadata": {},
+   "source": [
+    "## Running an Evaluation\n",
+    "\n",
+    "Running a scorebook evaluation with `evaluate` only requires an inference function and a dataset. When uploading results to Trismik's dashboard, an experiment and project id are also required. We can also specify in hyperparameters, which are passed to the inference function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "7031045c655d1062",
+   "metadata": {},
+   "source": [
+    "from scorebook import evaluate\n",
+    "\n",
+    "# Run evaluation\n",
+    "results = evaluate(\n",
+    "    inference=qwen,\n",
+    "    datasets = dataset,\n",
+    "    hyperparameters = {\n",
+    "        'temperature': 0.9,\n",
+    "        'top_p': 0.8,\n",
+    "        'top_k': 40,\n",
+    "        'system_message': \"Answer the question directly, provide no additional context.\"\n",
+    "    },\n",
+    "    experiment_id = \"Qwen-Classical-Evaluation\",\n",
+    "    project_id = TRISMIK_PROJECT_ID,\n",
+    ")\n",
+    "\n",
+    "print(\"Qwen2.5 0.5B Evaluation Results:\")\n",
+    "print(f\"accuracy: {results[0]['accuracy']}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "The results are encapsulated within a list of dictionaries, with a dict for each evaluation run. the above example only executes a single run as only 1 dataset, and hyperparameter configuration is evaluated.\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "- [Scorebook Docs](https://docs.trismik.com/scorebook/introduction-to-scorebook/): Scorebook's full documentation.\n",
+    "- [Scorebook Repository](https://github.com/trismik/scorebook): Scorebook is an open-source library, view the code and more examples."
+   ],
+   "id": "61d9ff67d63624d6"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

tutorials/quickstarts/getting_started.ipynb

@@ -0,0 +1,197 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bda7b5add96e4d97",
+   "metadata": {},
+   "source": [
+    "# Getting Started with Trismik's Adaptive Testing\n",
+    "\n",
+    "This notebook demonstrates how to run Trismik's adaptive evaluations using Scorebook.\n",
+    "\n",
+    "## What is Adaptive Testing?\n",
+    "\n",
+    "Trismik’s adaptive testing service leverages item response theory (IRT), a psychometric framework, to evaluate large language models. Using computerized adaptive testing (CAT) it dynamically selects the most informative items, enabling faster, more cost-efficient model evaluations with fewer items required.\n",
+    "\n",
+    "## Setup\n",
+    "\n",
+    "### Install Scorebook\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "286eca2349c6ddc6",
+   "metadata": {},
+   "source": "!pip install scorebook",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d2fa56528e14c46d",
+   "metadata": {},
+   "source": [
+    "\n",
+    "\n",
+    "### Generate a Trismik API Key\n",
+    "\n",
+    "To run an adaptive evaluation, a Trismik API key is required. You can [sign up](https://dashboard.trismik.com/signup) for a free Trismik account and generate an API key.\n",
+    "\n",
+    "**How to generate an API key from the Trismik dashboard**:\n",
+    "1. click on your initials in the top-right corner of the screen.\n",
+    "2. click on \"API Keys\" in the drop-down menu.\n",
+    "3. click \"Create API Key\" to create a new API key.\n",
+    "\n",
+    "### Set Trismik API Key"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "5ed8a62ac56560e9",
+   "metadata": {},
+   "source": [
+    "# Set your API key here and run this cell to login\n",
+    "TRISMIK_API_KEY = \"your-trismik-api-key-here\""
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "10d842e5b99e95bd",
+   "metadata": {},
+   "source": [
+    "### Login with Trismik API Key"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "initial_id",
+   "metadata": {},
+   "source": [
+    "import scorebook\n",
+    "\n",
+    "scorebook.login(TRISMIK_API_KEY)\n",
+    "print(\"✓ Logged in to Trismik\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eedbc41adba92248",
+   "metadata": {},
+   "source": [
+    "### Create a Project\n",
+    "\n",
+    "When running an adaptive evaluation, your evaluation results are stored under a project on the Trismik dashboard. Projects can be created from the dashboard's interface, or programmatically via Scorebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "bd7271800bf91478",
+   "metadata": {},
+   "source": [
+    "from scorebook import create_project\n",
+    "\n",
+    "# Create a project\n",
+    "project = create_project(\n",
+    "    name = \"Quick-Start Guides\",\n",
+    "    description = \"A project created for Trismik's quick-start guides.\"\n",
+    ")\n",
+    "\n",
+    "print(\"✓ Project created\")\n",
+    "print(f\"Project ID: {project.id}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e30ea2a2674e6005",
+   "metadata": {},
+   "source": [
+    "## Run an Adaptive Evaluation\n",
+    "\n",
+    "For this quick-start guide, we will use a mock model, that replicates the responses generated by an Amazon LLM (Nova-Pro). `mock_llm` is an inference function which accepts a list of model inputs and returns a list of model outputs for scoring. In this example, we use model responses that we pre-computed to showcase how adaptive evaluations work."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "d9f96d063e08c4fd",
+   "metadata": {},
+   "source": [
+    "from scorebook.utils.mock_llm import mock_llm\n",
+    "\n",
+    "# Run adaptive evaluation\n",
+    "results = scorebook.evaluate(\n",
+    "    inference = mock_llm,\n",
+    "    datasets = \"trismik/MMLUPro:adaptive\",\n",
+    "    experiment_id = \"Getting-Started\",\n",
+    "    project_id = project.id,\n",
+    ")\n",
+    "\n",
+    "# Print the adaptive evaluation results\n",
+    "print(\"✓ Adaptive evaluation complete!\")\n",
+    "print(\"Results: \", results[0][\"score\"])\n",
+    "\n",
+    "print(f\"You can view your results here: https://dashboard.trismik.com/projects/{project.id}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "63d3f34a76e90ac0",
+   "metadata": {},
+   "source": [
+    "### Adaptive Evaluation Results\n",
+    "\n",
+    "The metrics generated by an adaptive evaluation are:\n",
+    "\n",
+    "- Theta (θ): The primary score measuring model ability on the dataset, a higher value represents better performance.\n",
+    "- Standard Error: The theta score is a proxy for the underlying metric, and the standard error is the uncertainty in the theta estimate.\n",
+    "\n",
+    "You can find more information about adaptive testing [here](https://docs.trismik.com/adaptiveTesting/adaptive-testing-introduction/)!\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "**More Quick-Start Guides**:\n",
+    "\n",
+    "1. [HuggingFace Adaptive Evaluation](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/adaptive_evaluations/adaptive_evaluation_qwen_demo.ipynb): For a demo showcasing how Scorebook's adaptive evaluations can be used for local HF-based models. \n",
+    "2. [API-based Adaptive Evaluation](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/adaptive_evaluations/adaptive_evaluation_openai_demo.ipynb): For a demo showcasing how Scorebook's adaptive evaluations can be used for API-based models, using the OpenAI API.\n",
+    "3. [HuggingFace Full Dataset Evaluation](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/classical_evaluations/classical_evaluation_demo.ipynb): For a demo showcasing how Scorebook can be used for full dataset evaluations with familiar metrics, like `Accuracy`.\n",
+    "\n",
+    "**More details on Adaptive Testing and Scorebook**:\n",
+    "\n",
+    "- [Introduction to Adaptive Testing](https://docs.trismik.com/adaptiveTesting/adaptive-testing-introduction/): a quick introduction to Adaptive Testing.\n",
+    "- [Dataset Page](https://dashboard.trismik.com/datasets): Trismik's full set of currently adaptive datasets from the Trismik dashboard.\n",
+    "- [Scorebook Docs](https://docs.trismik.com/scorebook/introduction-to-scorebook/): Scorebook's full documentation.\n",
+    "- [Scorebook Repository](https://github.com/trismik/scorebook): Scorebook is an open-source library, view the code and more examples."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

tutorials/utils/__init__.py

@@ -0,0 +1,35 @@
+"""
+Helper utilities for Scorebook examples.
+
+This module provides common helper functions used across multiple Scorebook examples
+for setup, output handling, and argument parsing.
+"""
+
+# Argument parsing utilities
+from .args_parser import (
+    add_model_selection_arg,
+    create_parser,
+    parse_args_with_config,
+    setup_batch_model_parser,
+    setup_openai_model_parser,
+)
+
+# Output utilities
+from .output import save_results_to_json
+
+# Setup utilities
+from .setup import setup_logging, setup_output_directory
+
+__all__ = [
+    # Setup
+    "setup_logging",
+    "setup_output_directory",
+    # Output
+    "save_results_to_json",
+    # Argument parsing
+    "create_parser",
+    "add_model_selection_arg",
+    "setup_openai_model_parser",
+    "setup_batch_model_parser",
+    "parse_args_with_config",
+]

tutorials/utils/args_parser.py

@@ -0,0 +1,132 @@
+"""
+Generic argument parsing utilities for Scorebook examples.
+
+This module provides reusable argument parsing functions that can be used
+across multiple Scorebook examples for consistent command-line interfaces.
+"""
+
+import argparse
+from typing import Any, Dict, List, Optional
+
+
+def create_parser(description: str) -> argparse.ArgumentParser:
+    """Create a basic argument parser with a description.
+
+    Args:
+        description: Description for the argument parser
+
+    Returns:
+        Configured ArgumentParser instance
+    """
+    return argparse.ArgumentParser(description=description)
+
+
+def add_model_selection_arg(
+    parser: argparse.ArgumentParser,
+    default: str = "gpt-4o-mini",
+    help_text: Optional[str] = None,
+    supported_models: Optional[List[str]] = None,
+) -> argparse.ArgumentParser:
+    """Add model selection argument to parser.
+
+    Args:
+        parser: ArgumentParser to add the argument to
+        default: Default model name
+        help_text: Custom help text for the argument
+        supported_models: List of supported models for validation
+
+    Returns:
+        The modified parser
+    """
+    if help_text is None:
+        help_text = f"OpenAI model to use for inference (default: {default})"
+        if supported_models:
+            help_text += f". Supported models: {', '.join(supported_models)}"
+
+    parser.add_argument(
+        "--model",
+        type=str,
+        default=default,
+        help=help_text,
+    )
+    return parser
+
+
+def setup_openai_model_parser(
+    description: str = "Select OpenAI model for evaluation.",
+    default: str = "gpt-4o-mini",
+    supported_models: Optional[List[str]] = None,
+) -> str:
+    """Set up and parse OpenAI model selection arguments.
+
+    Args:
+        description: Description for the argument parser
+        default: Default model name
+        supported_models: List of supported models for help text
+
+    Returns:
+        Selected model name
+    """
+    parser = create_parser(description)
+    add_model_selection_arg(parser, default=default, supported_models=supported_models)
+    args = parser.parse_args()
+    return str(args.model)
+
+
+def setup_batch_model_parser(
+    description: str = "Select OpenAI model for batch evaluation.", default: str = "gpt-4o-mini"
+) -> str:
+    """Set up and parse OpenAI model selection arguments for batch inference.
+
+    Args:
+        description: Description for the argument parser
+        default: Default model name
+
+    Returns:
+        Selected model name
+    """
+    supported_models = ["gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "gpt-3.5-turbo"]
+    help_text = (
+        f"OpenAI model to use for batch inference. "
+        f"Note: Only select models support the Batch API. "
+        f"Supported models include: {', '.join(supported_models)}. "
+        f"Default: {default}"
+    )
+
+    parser = create_parser(description)
+    add_model_selection_arg(parser, default=default, help_text=help_text)
+    args = parser.parse_args()
+    return str(args.model)
+
+
+def parse_args_with_config(config: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:
+    """Parse arguments using a configuration dictionary.
+
+    Args:
+        config: Dictionary defining arguments to add. Format:
+            {
+                "arg_name": {
+                    "type": str,
+                    "default": "default_value",
+                    "help": "Help text",
+                    "required": False  # optional
+                }
+            }
+
+    Returns:
+        Dictionary of parsed argument values
+    """
+    parser = argparse.ArgumentParser()
+
+    for arg_name, arg_config in config.items():
+        kwargs = {"type": arg_config.get("type", str), "help": arg_config.get("help", "")}
+
+        if "default" in arg_config:
+            kwargs["default"] = arg_config["default"]
+        if "required" in arg_config:
+            kwargs["required"] = arg_config["required"]
+
+        parser.add_argument(f"--{arg_name.replace('_', '-')}", **kwargs)
+
+    args = parser.parse_args()
+    return vars(args)

tutorials/utils/output.py

@@ -0,0 +1,23 @@
+"""
+Utility functions for saving Scorebook evaluation results.
+
+This module provides common helper functions used across multiple Scorebook examples
+for saving evaluation results to files.
+"""
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+def save_results_to_json(results: Any, output_dir: Path, filename: str) -> None:
+    """Save evaluation results to a JSON file.
+
+    Args:
+        results: The evaluation results to save
+        output_dir: Directory to save the file in
+        filename: Name of the output file (should include .json extension)
+    """
+    output_path = output_dir / filename
+    with open(output_path, "w") as output_file:
+        json.dump(results, output_file, indent=4)