scorebook 0.0.14__py3-none-any.whl → 0.0.16__py3-none-any.whl

tutorials/notebooks/4-uploading_results.ipynb

@@ -0,0 +1,175 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "# Uploading Results to Trismik Dashboard\n\nThis notebook demonstrates three ways to upload evaluation results to Trismik's dashboard for tracking and visualization.\n\n## Why Upload Results?\n\n- **Track Progress**: Monitor model performance over time\n- **Compare Models**: Visualize performance across different models and experiments\n- **Share Results**: Collaborate with your team on evaluation insights\n- **Historical Analysis**: Maintain a record of all evaluations\n\n## Prerequisites\n\n- **Trismik API key**: Get yours at https://app.trismik.com/settings\n- **Trismik Project**: Create a project at https://app.trismik.com and copy its Project ID"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Setup Credentials\n\nSet your Trismik credentials here:"
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "# STEP 1: Get your Trismik API key from https://app.trismik.com/settings\n# STEP 2: Create a project at https://app.trismik.com and copy the Project ID\n\n# Set your credentials here\nTRISMIK_API_KEY = \"your-trismik-api-key\"\nTRISMIK_PROJECT_ID = \"your-project-id\"",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": [
+    "from pprint import pprint\n",
+    "from scorebook import score, login\n",
+    "from scorebook.metrics.accuracy import Accuracy"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Login to Trismik"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "if not TRISMIK_API_KEY or TRISMIK_API_KEY == \"your-trismik-api-key\":\n    raise ValueError(\"Please set TRISMIK_API_KEY. Get your API key from https://app.trismik.com/settings\")\n\nlogin(TRISMIK_API_KEY)\nprint(\"✓ Logged in to Trismik\")\n\nif not TRISMIK_PROJECT_ID or TRISMIK_PROJECT_ID == \"your-project-id\":\n    raise ValueError(\"Please set TRISMIK_PROJECT_ID. Create a project at https://app.trismik.com\")\n\nprint(f\"✓ Using project: {TRISMIK_PROJECT_ID}\")",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Method 1: Upload score() Results\n",
+    "\n",
+    "Score pre-computed outputs and upload to Trismik:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "# Prepare items with pre-computed outputs\nitems = [\n    {\"input\": \"What is 2 + 2?\", \"output\": \"4\", \"label\": \"4\"},\n    {\"input\": \"What is the capital of France?\", \"output\": \"Paris\", \"label\": \"Paris\"},\n    {\"input\": \"Who wrote Romeo and Juliet?\", \"output\": \"William Shakespeare\", \"label\": \"William Shakespeare\"},\n    {\"input\": \"What is 5 * 6?\", \"output\": \"30\", \"label\": \"30\"},\n    {\"input\": \"What is the largest planet?\", \"output\": \"Jupiter\", \"label\": \"Jupiter\"},\n]\n\n# Score and upload\nresults = score(\n    items=items,\n    metrics=Accuracy,\n    dataset_name=\"basic_questions\",\n    model_name=\"example-model-v1\",\n    experiment_id=\"Score-Upload-Notebook\",\n    project_id=TRISMIK_PROJECT_ID,\n    metadata={\n        \"description\": \"Example from Jupyter notebook\",\n        \"note\": \"Pre-computed outputs uploaded via score()\",\n    },\n    upload_results=True,  # Enable uploading\n)\n\nprint(f\"\\n✓ Results uploaded successfully!\")\nprint(f\"Accuracy: {results['aggregate_results'][0]['accuracy']:.2%}\")",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Method 2: Upload evaluate() Results\n",
+    "\n",
+    "Run inference and automatically upload results:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "from typing import Any, List\nfrom scorebook import EvalDataset, evaluate\n\n# Create a simple dataset\nimport json\nfrom pathlib import Path\n\nsample_data = [\n    {\"question\": \"What is 10 + 5?\", \"answer\": \"15\"},\n    {\"question\": \"What is the capital of Spain?\", \"answer\": \"Madrid\"},\n]\n\ntemp_file = Path(\"temp_eval_dataset.json\")\nwith open(temp_file, \"w\") as f:\n    json.dump(sample_data, f)\n\ndataset = EvalDataset.from_json(\n    path=str(temp_file),\n    metrics=\"accuracy\",\n    input=\"question\",\n    label=\"answer\",\n)\n\n# Define a simple inference function (mock)\ndef mock_inference(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n    \"\"\"Mock inference that returns the expected answers.\"\"\"\n    # In practice, this would call your model\n    return [\"15\", \"Madrid\"]  # Mock perfect answers\n\n# Run evaluation with upload\neval_results = evaluate(\n    mock_inference,\n    dataset,\n    hyperparameters={\"temperature\": 0.7},\n    experiment_id=\"Evaluate-Upload-Notebook\",\n    project_id=TRISMIK_PROJECT_ID,\n    metadata={\n        \"model\": \"mock-model\",\n        \"description\": \"Evaluation results from notebook\",\n    },\n    return_aggregates=True,\n    return_items=True,\n    return_output=True,\n)\n\nprint(f\"\\n✓ Evaluation results uploaded!\")\nprint(f\"Accuracy: {eval_results['aggregate_results'][0]['accuracy']:.2%}\")\n\n# Cleanup\ntemp_file.unlink()",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Method 3: Upload External Results\n",
+    "\n",
+    "Import results from external evaluation frameworks or historical data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "# Example: Import results from another evaluation framework\nexternal_results = [\n    {\"input\": \"Translate 'hello' to Spanish\", \"output\": \"hola\", \"label\": \"hola\"},\n    {\"input\": \"Translate 'goodbye' to Spanish\", \"output\": \"adiós\", \"label\": \"adiós\"},\n    {\"input\": \"Translate 'thank you' to Spanish\", \"output\": \"gracias\", \"label\": \"gracias\"},\n    {\"input\": \"Translate 'please' to Spanish\", \"output\": \"por favor\", \"label\": \"por favor\"},\n]\n\n# Upload external results\nexternal_upload = score(\n    items=external_results,\n    metrics=\"accuracy\",\n    dataset_name=\"spanish_translation\",\n    model_name=\"external-translator-v2\",\n    experiment_id=\"External-Results-Upload\",\n    project_id=TRISMIK_PROJECT_ID,\n    metadata={\n        \"description\": \"Historical results imported from external framework\",\n        \"source\": \"Custom evaluation pipeline\",\n        \"date\": \"2025-01-15\",\n    },\n    upload_results=True,\n)\n\nprint(f\"\\n✓ External results uploaded!\")\nprint(f\"Accuracy: {external_upload['aggregate_results'][0]['accuracy']:.2%}\")",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## View Results on Dashboard\n",
+    "\n",
+    "All uploaded results are now visible on your Trismik dashboard:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "from IPython.display import display, Markdown\n\ndashboard_url = f\"https://app.trismik.com/projects/{TRISMIK_PROJECT_ID}\"\ndisplay(Markdown(f\"### 📊 [View All Results on Dashboard]({dashboard_url})\"))\nprint(f\"\\nDirect link: {dashboard_url}\")\nprint(\"\\nYou should see three experiments:\")\nprint(\"  1. Score-Upload-Notebook\")\nprint(\"  2. Evaluate-Upload-Notebook\")\nprint(\"  3. External-Results-Upload\")",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Organizing Results with Metadata\n",
+    "\n",
+    "Use metadata to add context and organization to your results:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "source": "# Example: Organizing a model comparison experiment\nmodels_to_test = [\n    {\"name\": \"model-a\", \"version\": \"1.0\"},\n    {\"name\": \"model-b\", \"version\": \"2.0\"},\n]\n\ntest_items = [\n    {\"output\": \"positive\", \"label\": \"positive\"},\n    {\"output\": \"negative\", \"label\": \"negative\"},\n]\n\nfor model_info in models_to_test:\n    result = score(\n        items=test_items,\n        metrics=Accuracy,\n        dataset_name=\"sentiment_test\",\n        model_name=model_info[\"name\"],\n        experiment_id=\"Model-Comparison-Notebook\",\n        project_id=TRISMIK_PROJECT_ID,\n        metadata={\n            \"model_version\": model_info[\"version\"],\n            \"comparison_group\": \"sentiment_analysis\",\n            \"date\": \"2025-01-26\",\n            \"notes\": f\"Testing {model_info['name']} v{model_info['version']}\",\n        },\n        upload_results=True,\n    )\n    print(f\"✓ Uploaded results for {model_info['name']} v{model_info['version']}\")",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Best Practices\n",
+    "\n",
+    "### Experiment Naming\n",
+    "- Use descriptive `experiment_id` values (e.g., \"GPT4-MMLU-Baseline\")\n",
+    "- Group related runs under the same experiment ID\n",
+    "- Use different experiment IDs for different types of tests\n",
+    "\n",
+    "### Metadata\n",
+    "- Include model version, hyperparameters, and configuration\n",
+    "- Add timestamps and descriptions for historical tracking\n",
+    "- Use consistent keys across experiments for easy comparison\n",
+    "\n",
+    "### Organization\n",
+    "- Create separate projects for different use cases\n",
+    "- Use tags or metadata fields to categorize experiments\n",
+    "- Document your evaluation methodology in metadata\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "- Explore the Trismik dashboard to visualize trends and comparisons\n",
+    "- Set up automated evaluation pipelines with result uploading\n",
+    "- Try the **Adaptive Evaluations** notebook for efficient testing with automatic uploads"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

tutorials/quickstarts/adaptive_evaluations/adaptive_evaluation_openai_demo.ipynb

@@ -0,0 +1,229 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bc3ba3cd77800bb4",
+   "metadata": {},
+   "source": [
+    "# Adaptive Evaluations with Scorebook - Evaluating an OpenAI GPT Model\n",
+    "\n",
+    "This quick-start guide showcases an adaptive evaluation of OpenAI's GPT-4o Mini model.\n",
+    "\n",
+    "We recommend that you first see our [getting started quick-start guide](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/getting_started.ipynb) if you have not done so already, for more of a detailed overview on adaptive testing and setting up Trismik credentials.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "- **Trismik API key**: Generate a Trismik API key from the [Trismik dashboard's settings page](https://app.trismik.com/settings).\n",
+    "- **Trismik Project Id**: We recommend you use the project id generated in the [Getting Started Quick-Start Guide](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/getting_started.ipynb).\n",
+    "- **OpenAI API key**: Generate an OpenAI API key from [OpenAI's API Platform](https://openai.com/api/).\n",
+    "\n",
+    "## Install Scorebook"
+   ]
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "source": [
+    "!pip install scorebook\n",
+    "# if you're running this locally, please run !pip install scorebook\"[examples, providers]\""
+   ],
+   "id": "f454e876551a4a0c",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "\n",
+    "## Setup Credentials\n",
+    "\n",
+    "Enter your Trismik API key, project id and OpenAI API Key below."
+   ],
+   "id": "cad992b287d4d0ac"
+  },
+  {
+   "cell_type": "code",
+   "id": "14e576282749edb7",
+   "metadata": {},
+   "source": [
+    "# Set your credentials here\n",
+    "TRISMIK_API_KEY = \"your-trismik-api-key-here\"\n",
+    "TRISMIK_PROJECT_ID = \"your-trismik-project-id-here\"\n",
+    "OPENAI_API_KEY = \"your-openai-api-key-here\""
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "700950d039e4c0f6",
+   "metadata": {},
+   "source": [
+    "## Login with Trismik API Key"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "initial_id",
+   "metadata": {},
+   "source": [
+    "from scorebook import login\n",
+    "\n",
+    "# Login to Trismik\n",
+    "login(TRISMIK_API_KEY)\n",
+    "print(\"✓ Logged in to Trismik\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13084db21e549ccf",
+   "metadata": {},
+   "source": [
+    "## Define an Inference Function\n",
+    "\n",
+    "To evaluate a model with Scorebook, it must be encapsulated within an inference function. An inference function must accept a list of model inputs, pass these to the model for inference, collect and return outputs generated.\n",
+    "\n",
+    "An inference function can be defined to encapsulate any model, local or cloud-hosted. There is flexibility in how an inference function can be defined, the only requirements are the function signature. An inference function must,\n",
+    "\n",
+    "Accept:\n",
+    "\n",
+    "- A list of model inputs.\n",
+    "- Hyperparameters which can be optionally accessed via kwargs.\n",
+    "\n",
+    "Return\n",
+    "\n",
+    "- A list of parsed model outputs for scoring."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "8aa99f513db6241a",
+   "metadata": {},
+   "source": [
+    "from openai import OpenAI\n",
+    "from typing import Any, List\n",
+    "import string\n",
+    "\n",
+    "client = OpenAI(api_key=OPENAI_API_KEY)\n",
+    "\n",
+    "# define an inference function for GPT-4o Mini.\n",
+    "def gpt4o_mini(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Process inputs through OpenAI's API\"\"\"\n",
+    "\n",
+    "    outputs = []\n",
+    "    for idx, input_item in enumerate(inputs):\n",
+    "\n",
+    "        # Format prompt\n",
+    "        choices = input_item.get(\"options\", [])\n",
+    "        prompt = (\n",
+    "            str(input_item.get(\"question\", \"\"))\n",
+    "            + \"\\nOptions:\\n\"\n",
+    "            + \"\\n\".join(\n",
+    "                f\"{letter}: {choice['text'] if isinstance(choice, dict) else choice}\"\n",
+    "                for letter, choice in zip(string.ascii_uppercase, choices)\n",
+    "            )\n",
+    "        )\n",
+    "\n",
+    "        # Build messages for OpenAI API\n",
+    "        messages = [\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": hyperparameters[\"system_message\"]\n",
+    "            },\n",
+    "            {\"role\": \"user\", \"content\": prompt},\n",
+    "        ]\n",
+    "\n",
+    "        # Call OpenAI API and extract output from the response\n",
+    "        try:\n",
+    "            response = client.chat.completions.create(\n",
+    "                model=\"gpt-4o-mini\",\n",
+    "                messages=messages,\n",
+    "                temperature=0.7,\n",
+    "            )\n",
+    "            output = response.choices[0].message.content.strip()\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            output = f\"Error: {str(e)}\"\n",
+    "\n",
+    "        outputs.append(output)\n",
+    "\n",
+    "    return outputs"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "efa5c3ea791bbcd1",
+   "metadata": {},
+   "source": [
+    "## Run an Adaptive Evaluation\n",
+    "\n",
+    "When running an adaptive evaluation, we can use any single or multiple adaptive datasets and specify a split to be evaluated."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "3cbf1b2f13d5553e",
+   "metadata": {},
+   "source": [
+    "from scorebook import evaluate\n",
+    "\n",
+    "# Run adaptive evaluation\n",
+    "results = evaluate(\n",
+    "    inference = gpt4o_mini,\n",
+    "    datasets = \"trismik/figQA:adaptive\",\n",
+    "    hyperparameters = {\"system_message\": \"Answer the question with only the letter of the correct option. No additional text or context\"},\n",
+    "    split = \"validation\",\n",
+    "    experiment_id = \"GPT-4o-Mini-Adaptive-Evaluation\",\n",
+    "    project_id = TRISMIK_PROJECT_ID,\n",
+    ")\n",
+    "\n",
+    "# Print the adaptive evaluation results\n",
+    "print(\"✓ Adaptive evaluation complete!\")\n",
+    "print(\"Results: \", results[0][\"score\"])"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d37cb5e87cc297fe",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "- [Adaptive Testing White Paper](https://docs.trismik.com/adaptiveTesting/adaptive-testing-introduction/): An in depth overview of the science behind the adaptive testing methodology.\n",
+    "- [Dataset Page](https://dashboard.trismik.com/datasets): Trismik's full set of currently adaptive datasets from the Trismik dashboard.\n",
+    "- [Scorebook Docs](https://docs.trismik.com/scorebook/introduction-to-scorebook/): Scorebook's full documentation.\n",
+    "- [Scorebook Repository](https://github.com/trismik/scorebook): Scorebook is an open-source library, view the code and more examples."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

tutorials/quickstarts/adaptive_evaluations/adaptive_evaluation_qwen_demo.ipynb

@@ -0,0 +1,256 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bc3ba3cd77800bb4",
+   "metadata": {},
+   "source": [
+    "# Adaptive Evaluations with Scorebook - Evaluating a Local Qwen Model\n",
+    "\n",
+    "This quick-start guide showcases an adaptive evaluation of Qwen's Qwen2.5 0.5B Instruct model.\n",
+    "\n",
+    "We recommend that you first see our [getting started quick-start guide](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/getting_started.ipynb) if you have not done so already, for more of a detailed overview on adaptive testing and setting up Trismik credentials.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "- **Trismik API key**: Generate a Trismik API key from the [Trismik dashboard's settings page](https://app.trismik.com/settings).\n",
+    "- **Trismik Project Id**: We recommend you use the project id generated in the [Getting Started Quick-Start Guide](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/getting_started.ipynb).\n",
+    "\n",
+    "## Install Scorebook"
+   ]
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "source": [
+    "!pip install scorebook\n",
+    "# if you're running this locally, please run !pip install scorebook\"[examples]\""
+   ],
+   "id": "90146caef86f19ee",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cad992b287d4d0ac",
+   "metadata": {},
+   "source": [
+    "## Setup Credentials\n",
+    "\n",
+    "Enter your Trismik API key and project id below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "14e576282749edb7",
+   "metadata": {},
+   "source": [
+    "# Set your credentials here\n",
+    "TRISMIK_API_KEY = \"your-trismik-api-key-here\"\n",
+    "TRISMIK_PROJECT_ID = \"your-trismik-project-id-here\""
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "700950d039e4c0f6",
+   "metadata": {},
+   "source": [
+    "## Login with Trismik API Key"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "initial_id",
+   "metadata": {},
+   "source": [
+    "from scorebook import login\n",
+    "\n",
+    "# Login to Trismik\n",
+    "login(TRISMIK_API_KEY)\n",
+    "print(\"✓ Logged in to Trismik\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "609a95a43d8cfc2c",
+   "metadata": {},
+   "source": [
+    "## Instantiate a Local Qwen Model\n",
+    "\n",
+    "For this quick-start guide, we will use the lightweight Qwen2.5 0.5B instruct model, via Hugging Face's transformers package."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "d1da8af72ef8de6f",
+   "metadata": {},
+   "source": [
+    "import transformers\n",
+    "\n",
+    "# Instantiate a model\n",
+    "pipeline = transformers.pipeline(\n",
+    "    \"text-generation\",\n",
+    "    model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n",
+    "    model_kwargs={\"torch_dtype\": \"auto\"},\n",
+    "    device_map=\"auto\",\n",
+    ")\n",
+    "\n",
+    "print(\"✓ Transformers pipeline instantiated\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13084db21e549ccf",
+   "metadata": {},
+   "source": [
+    "## Define an Inference Function\n",
+    "\n",
+    "To evaluate a model with Scorebook, it must be encapsulated within an inference function. An inference function must accept a list of model inputs, pass these to the model for inference, collect and return outputs generated.\n",
+    "\n",
+    "An inference function can be defined to encapsulate any model, local or cloud-hosted. There is flexibility in how an inference function can be defined, the only requirements are the function signature. An inference function must,\n",
+    "\n",
+    "Accept:\n",
+    "\n",
+    "- A list of model inputs.\n",
+    "- Hyperparameters which can be optionally accessed via kwargs.\n",
+    "\n",
+    "Return\n",
+    "\n",
+    "- A list of parsed model outputs for scoring."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "8aa99f513db6241a",
+   "metadata": {},
+   "source": [
+    "from typing import Any, List\n",
+    "import string\n",
+    "\n",
+    "# Define an inference function for the Qwen model.\n",
+    "def qwen(inputs: List[Any], **hyperparameters: Any) -> List[Any]:\n",
+    "    \"\"\"Process inputs through Qwen model\"\"\"\n",
+    "\n",
+    "    outputs = []\n",
+    "    for idx, input_item in enumerate(inputs):\n",
+    "\n",
+    "        # Format prompt\n",
+    "        choices = input_item.get(\"options\", [])\n",
+    "        prompt = (\n",
+    "            str(input_item.get(\"question\", \"\"))\n",
+    "            + \"\\nOptions:\\n\"\n",
+    "            + \"\\n\".join(\n",
+    "                f\"{letter}: {choice['text'] if isinstance(choice, dict) else choice}\"\n",
+    "                for letter, choice in zip(string.ascii_uppercase, choices)\n",
+    "            )\n",
+    "        )\n",
+    "\n",
+    "        # Build messages for Qwen model\n",
+    "        messages = [\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": hyperparameters[\"system_message\"]\n",
+    "            },\n",
+    "            {\"role\": \"user\", \"content\": prompt},\n",
+    "        ]\n",
+    "\n",
+    "        # Run inference using the pipeline\n",
+    "        try:\n",
+    "            output = pipeline(\n",
+    "                messages,\n",
+    "                temperature = hyperparameters.get(\"temperature\", 0.7),\n",
+    "                top_p = hyperparameters.get(\"top_p\", 0.9),\n",
+    "                top_k = hyperparameters.get(\"top_k\", 50),\n",
+    "                max_new_tokens = 512,\n",
+    "                do_sample = hyperparameters.get(\"temperature\", 0.7) > 0,\n",
+    "            )\n",
+    "            response = output[0][\"generated_text\"][-1][\"content\"].strip()\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            response = f\"Error: {str(e)}\"\n",
+    "\n",
+    "        outputs.append(response)\n",
+    "\n",
+    "    return outputs"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "efa5c3ea791bbcd1",
+   "metadata": {},
+   "source": [
+    "## Run an Adaptive Evaluation\n",
+    "\n",
+    "When running an adaptive evaluation, we can use any single or multiple adaptive datasets and specify a split to be evaluated."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "3cbf1b2f13d5553e",
+   "metadata": {},
+   "source": [
+    "from scorebook import evaluate\n",
+    "\n",
+    "# Run adaptive evaluation\n",
+    "results = evaluate(\n",
+    "    inference = qwen,\n",
+    "    datasets = \"trismik/figQA:adaptive\",\n",
+    "    hyperparameters = {\"system_message\": \"Answer the question with only the letter of the correct option. No additional text or context\"},\n",
+    "    split = \"validation\",\n",
+    "    experiment_id = \"Qwen-2.5-0.5B-Adaptive-Evaluation\",\n",
+    "    project_id = TRISMIK_PROJECT_ID,\n",
+    ")\n",
+    "\n",
+    "# Print the adaptive evaluation results\n",
+    "print(\"✓ Adaptive evaluation complete!\")\n",
+    "print(\"Results: \", results[0][\"score\"])"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d37cb5e87cc297fe",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "- [Adaptive Testing White Paper](https://docs.trismik.com/adaptiveTesting/adaptive-testing-introduction/): An in depth overview of the science behind the adaptive testing methodology.\n",
+    "- [Dataset Page](https://dashboard.trismik.com/datasets): Trismik's full set of currently adaptive datasets from the Trismik dashboard.\n",
+    "- [Scorebook Docs](https://docs.trismik.com/scorebook/introduction-to-scorebook/): Scorebook's full documentation.\n",
+    "- [Scorebook Repository](https://github.com/trismik/scorebook): Scorebook is an open-source library, view the code and more examples."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}