scorebook 0.0.13__py3-none-any.whl → 0.0.15__py3-none-any.whl

tutorials/quickstarts/adaptive_evaluations/adaptive_evaluation_qwen_demo.ipynb

@@ -0,0 +1,256 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bc3ba3cd77800bb4",
+   "metadata": {},
+   "source": [
+    "# Adaptive Evaluations with Scorebook - Evaluating a Local Qwen Model\n",
+    "\n",
+    "This quick-start guide showcases an adaptive evaluation of Qwen's Qwen2.5 0.5B Instruct model.\n",
+    "\n",
+    "We recommend that you first see our [getting started quick-start guide](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/getting_started.ipynb) if you have not done so already, for more of a detailed overview on adaptive testing and setting up Trismik credentials.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "- **Trismik API key**: Generate a Trismik API key from the [Trismik dashboard's settings page](https://app.trismik.com/settings).\n",
+    "- **Trismik Project Id**: We recommend you use the project id generated in the [Getting Started Quick-Start Guide](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/getting_started.ipynb).\n",
+    "\n",
+    "## Install Scorebook"
+   ]
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "source": [
+    "!pip install scorebook\n",
+    "# if you're running this locally, please run !pip install scorebook\"[examples]\""
+   ],
+   "id": "90146caef86f19ee",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cad992b287d4d0ac",
+   "metadata": {},
+   "source": [
+    "## Setup Credentials\n",
+    "\n",
+    "Enter your Trismik API key and project id below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "14e576282749edb7",
+   "metadata": {},
+   "source": [
+    "# Set your credentials here\n",
+    "TRISMIK_API_KEY = \"your-trismik-api-key-here\"\n",
+    "TRISMIK_PROJECT_ID = \"your-trismik-project-id-here\""
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "700950d039e4c0f6",
+   "metadata": {},
+   "source": [
+    "## Login with Trismik API Key"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "initial_id",
+   "metadata": {},
+   "source": [
+    "from scorebook import login\n",
+    "\n",
+    "# Login to Trismik\n",
+    "login(TRISMIK_API_KEY)\n",
+    "print(\"✓ Logged in to Trismik\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "609a95a43d8cfc2c",
+   "metadata": {},
+   "source": [
+    "## Instantiate a Local Qwen Model\n",
+    "\n",
+    "For this quick-start guide, we will use the lightweight Qwen2.5 0.5B instruct model, via Hugging Face's transformers package."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "d1da8af72ef8de6f",
+   "metadata": {},
+   "source": [
+    "import transformers\n",
+    "\n",
+    "# Instantiate a model\n",
+    "pipeline = transformers.pipeline(\n",
+    "    \"text-generation\",\n",
+    "    model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n",
+    "    model_kwargs={\"torch_dtype\": \"auto\"},\n",
+    "    device_map=\"auto\",\n",
+    ")\n",
+    "\n",
+    "print(\"✓ Transformers pipeline instantiated\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13084db21e549ccf",
+   "metadata": {},
+   "source": [
+    "## Define an Inference Function\n",
+    "\n",
+    "To evaluate a model with Scorebook, it must be encapsulated within an inference function. An inference function must accept a list of model inputs, pass these to the model for inference, collect and return outputs generated.\n",
+    "\n",
+    "An inference function can be defined to encapsulate any model, local or cloud-hosted. There is flexibility in how an inference function can be defined, the only requirements are the function signature. An inference function must,\n",
+    "\n",
+    "Accept:\n",
+    "\n",
+    "- A list of model inputs.\n",
+    "- Hyperparameters which can be optionally accessed via kwargs.\n",
+    "\n",
+    "Return\n",
+    "\n",
+    "- A list of parsed model outputs for scoring."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "8aa99f513db6241a",
+   "metadata": {},
+   "source": [
+    "from typing import Any, List\n",
+    "import string\n",
+    "\n",
+    "# Define an inference function for the Qwen model.\n",
+    "def qwen(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Process inputs through Qwen model\"\"\"\n",
+    "\n",
+    "    outputs = []\n",
+    "    for idx, input_item in enumerate(inputs):\n",
+    "\n",
+    "        # Format prompt\n",
+    "        choices = input_item.get(\"options\", [])\n",
+    "        prompt = (\n",
+    "            str(input_item.get(\"question\", \"\"))\n",
+    "            + \"\\nOptions:\\n\"\n",
+    "            + \"\\n\".join(\n",
+    "                f\"{letter}: {choice['text'] if isinstance(choice, dict) else choice}\"\n",
+    "                for letter, choice in zip(string.ascii_uppercase, choices)\n",
+    "            )\n",
+    "        )\n",
+    "\n",
+    "        # Build messages for Qwen model\n",
+    "        messages = [\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": hyperparameters[\"system_message\"]\n",
+    "            },\n",
+    "            {\"role\": \"user\", \"content\": prompt},\n",
+    "        ]\n",
+    "\n",
+    "        # Run inference using the pipeline\n",
+    "        try:\n",
+    "            output = pipeline(\n",
+    "                messages,\n",
+    "                temperature = hyperparameters.get(\"temperature\", 0.7),\n",
+    "                top_p = hyperparameters.get(\"top_p\", 0.9),\n",
+    "                top_k = hyperparameters.get(\"top_k\", 50),\n",
+    "                max_new_tokens = 512,\n",
+    "                do_sample = hyperparameters.get(\"temperature\", 0.7) > 0,\n",
+    "            )\n",
+    "            response = output[0][\"generated_text\"][-1][\"content\"].strip()\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            response = f\"Error: {str(e)}\"\n",
+    "\n",
+    "        outputs.append(response)\n",
+    "\n",
+    "    return outputs"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "efa5c3ea791bbcd1",
+   "metadata": {},
+   "source": [
+    "## Run an Adaptive Evaluation\n",
+    "\n",
+    "When running an adaptive evaluation, we can use any single or multiple adaptive datasets and specify a split to be evaluated."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "3cbf1b2f13d5553e",
+   "metadata": {},
+   "source": [
+    "from scorebook import evaluate\n",
+    "\n",
+    "# Run adaptive evaluation\n",
+    "results = evaluate(\n",
+    "    inference = qwen,\n",
+    "    datasets = \"trismik/figQA:adaptive\",\n",
+    "    hyperparameters = {\"system_message\": \"Answer the question with only the letter of the correct option. No additional text or context\"},\n",
+    "    split = \"validation\",\n",
+    "    experiment_id = \"Qwen-2.5-0.5B-Adaptive-Evaluation\",\n",
+    "    project_id = TRISMIK_PROJECT_ID,\n",
+    ")\n",
+    "\n",
+    "# Print the adaptive evaluation results\n",
+    "print(\"✓ Adaptive evaluation complete!\")\n",
+    "print(\"Results: \", results[0][\"score\"])"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d37cb5e87cc297fe",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "- [Adaptive Testing White Paper](https://docs.trismik.com/adaptiveTesting/adaptive-testing-introduction/): An in depth overview of the science behind the adaptive testing methodology.\n",
+    "- [Dataset Page](https://dashboard.trismik.com/datasets): Trismik's full set of currently adaptive datasets from the Trismik dashboard.\n",
+    "- [Scorebook Docs](https://docs.trismik.com/scorebook/introduction-to-scorebook/): Scorebook's full documentation.\n",
+    "- [Scorebook Repository](https://github.com/trismik/scorebook): Scorebook is an open-source library, view the code and more examples."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

tutorials/quickstarts/classical_evaluations/classical_evaluation_demo.ipynb

@@ -0,0 +1,277 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "b1ef68792a1c82f2",
+   "metadata": {},
+   "source": [
+    "# _Classical_ Evaluations with Scorebook\n",
+    "\n",
+    "Scorebook, developed by Trismik, is an open-source Python library for model evaluation. It supports both Trismik’s adaptive testing and traditional classical evaluations. In a classical evaluation, a model runs inference on every item in a dataset, and the results are scored using Scorebook’s built-in metrics, such as accuracy, to produce evaluation results. Evaluation results can be automatically uploaded to the Scorebook dashboard, organized by project, for storing, managing, and visualizing model evaluation experiments.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "- **Trismik API key**: Generate a Trismik API key from the [Trismik dashboard's settings page](https://app.trismik.com/settings).\n",
+    "- **Trismik Project Id**: We recommend you use the project id generated in the [Getting Started Quick-Start Guide](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/getting_started.ipynb).\n",
+    "\n",
+    "### Install Scorebook\n"
+   ]
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "source": "!pip install scorebook",
+   "id": "c2fecbb3d7b699b3",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "\n",
+    "\n",
+    "### Setup Credentials\n",
+    "\n",
+    "Enter your Trismik API key and project id below."
+   ],
+   "id": "8e18d0e628fdb944"
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "source": [
+    "# Set your credentials here\n",
+    "TRISMIK_API_KEY = \"your-trismik-api-key-here\"\n",
+    "TRISMIK_PROJECT_ID = \"your-trismik-project-id-key-here\""
+   ],
+   "id": "4cb833e7b092ae4",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "### Login with Trismik API Key",
+   "id": "ed4e8281ecb99685"
+  },
+  {
+   "cell_type": "code",
+   "id": "f59fb493a575d361",
+   "metadata": {},
+   "source": [
+    "from scorebook import login\n",
+    "\n",
+    "login(TRISMIK_API_KEY)\n",
+    "print(\"✓ Logged in to Trismik\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cb58f57296229115",
+   "metadata": {},
+   "source": [
+    "\n",
+    "\n",
+    "## Evaluation Datasets\n",
+    "\n",
+    "A scorebook evaluation requires an evaluation dataset, represented by the `EvalDataset` class. Evaluation datasets can be constructed via a number of factory methods. In this example we will create a basic evaluation dataset from a list of evaluation items."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "fa7c936b75c83cad",
+   "metadata": {},
+   "source": [
+    "from scorebook import EvalDataset\n",
+    "from scorebook.metrics.accuracy import Accuracy\n",
+    "\n",
+    "# Create a sample dataset from a list of multiple-choice questions\n",
+    "evaluation_items = [\n",
+    "    {\"question\": \"What is 2 + 2?\", \"answer\": \"4\"},\n",
+    "    {\"question\": \"What is the capital of France?\", \"answer\": \"Paris\"},\n",
+    "    {\"question\": \"Who wrote Romeo and Juliet?\", \"answer\": \"William Shakespeare\"},\n",
+    "    {\"question\": \"What is the chemical symbol for gold?\", \"answer\": \"Au\"}\n",
+    "]\n",
+    "\n",
+    "# Create an EvalDataset from the list\n",
+    "dataset = EvalDataset.from_list(\n",
+    "    name = \"sample_multiple_choice\",\n",
+    "    metrics = Accuracy,\n",
+    "    items = evaluation_items,\n",
+    "    input = \"question\",\n",
+    "    label = \"answer\",\n",
+    ")\n",
+    "\n",
+    "print(f\"✓ Created dataset with {len(dataset.items)} items\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "609a95a43d8cfc2c",
+   "metadata": {},
+   "source": [
+    "## Preparing Models for Evaluation\n",
+    "\n",
+    "To evaluate a model with Scorebook, it must be encapsulated within an inference function. An inference function must accept a list of model inputs, pass these to the model for inference, collect and return outputs generated.\n",
+    "\n",
+    "### Instantiate a Local Qwen Model\n",
+    "\n",
+    "For this quick-start guide, we will use the lightweight Qwen2.5 0.5B instruct model, via Hugging Face's transformers package."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "d1da8af72ef8de6f",
+   "metadata": {},
+   "source": [
+    "import transformers\n",
+    "\n",
+    "# Instantiate a model\n",
+    "pipeline = transformers.pipeline(\n",
+    "    \"text-generation\",\n",
+    "    model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n",
+    "    model_kwargs={\"torch_dtype\": \"auto\"},\n",
+    "    device_map=\"auto\",\n",
+    ")\n",
+    "\n",
+    "print(\"✓ Transformers pipeline instantiated\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3b56a72374920220",
+   "metadata": {},
+   "source": [
+    "### Define an Inference Function\n",
+    "\n",
+    "An inference function can be defined to encapsulate any model, local or cloud-hosted. There is flexibility in how an inference function can be defined, the only requirements are the function signature. An inference function must,\n",
+    "\n",
+    "Accept:\n",
+    "\n",
+    "- A list of model inputs.\n",
+    "- Hyperparameters which can be optionally accessed via kwargs.\n",
+    "\n",
+    "Return\n",
+    "\n",
+    "- A list of parsed model outputs for scoring.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "55f6d1eee2fa886e",
+   "metadata": {},
+   "source": [
+    "from typing import Any, List\n",
+    "\n",
+    "# Define an inference function for the Qwen model.\n",
+    "def qwen(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Run inference on a list of inputs using the 0.5B Qwen model.\"\"\"\n",
+    "    inference_outputs = []\n",
+    "\n",
+    "    for model_input in inputs:\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": hyperparameters.get(\"system_message\", \"You are a helpful assistant.\")},\n",
+    "            {\"role\": \"user\", \"content\": str(model_input)},\n",
+    "        ]\n",
+    "\n",
+    "        output = pipeline(\n",
+    "            messages,\n",
+    "            temperature = hyperparameters.get(\"temperature\", 0.7),\n",
+    "            top_p = hyperparameters.get(\"top_p\", 0.9),\n",
+    "            top_k = hyperparameters.get(\"top_k\", 50),\n",
+    "            max_new_tokens = 512,\n",
+    "            do_sample = hyperparameters.get(\"temperature\", 0.7) > 0,\n",
+    "        )\n",
+    "\n",
+    "        inference_outputs.append(output[0][\"generated_text\"][-1][\"content\"])\n",
+    "\n",
+    "    return inference_outputs\n",
+    "\n",
+    "print(\"✓ Inference function for Qwen2.5 0.5B defined\")\n",
+    "print(qwen([\"Hello!\"]))"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1230b9ad762482af",
+   "metadata": {},
+   "source": [
+    "## Running an Evaluation\n",
+    "\n",
+    "Running a scorebook evaluation with `evaluate` only requires an inference function and a dataset. When uploading results to Trismik's dashboard, an experiment and project id are also required. We can also specify in hyperparameters, which are passed to the inference function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "7031045c655d1062",
+   "metadata": {},
+   "source": [
+    "from scorebook import evaluate\n",
+    "\n",
+    "# Run evaluation\n",
+    "results = evaluate(\n",
+    "    inference=qwen,\n",
+    "    datasets = dataset,\n",
+    "    hyperparameters = {\n",
+    "        'temperature': 0.9,\n",
+    "        'top_p': 0.8,\n",
+    "        'top_k': 40,\n",
+    "        'system_message': \"Answer the question directly, provide no additional context.\"\n",
+    "    },\n",
+    "    experiment_id = \"Qwen-Classical-Evaluation\",\n",
+    "    project_id = TRISMIK_PROJECT_ID,\n",
+    ")\n",
+    "\n",
+    "print(\"Qwen2.5 0.5B Evaluation Results:\")\n",
+    "print(f\"accuracy: {results[0]['accuracy']}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "The results are encapsulated within a list of dictionaries, with a dict for each evaluation run. the above example only executes a single run as only 1 dataset, and hyperparameter configuration is evaluated.\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "- [Scorebook Docs](https://docs.trismik.com/scorebook/introduction-to-scorebook/): Scorebook's full documentation.\n",
+    "- [Scorebook Repository](https://github.com/trismik/scorebook): Scorebook is an open-source library, view the code and more examples."
+   ],
+   "id": "61d9ff67d63624d6"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}