scorebook 0.0.13__py3-none-any.whl → 0.0.15__py3-none-any.whl

tutorials/quickstarts/getting_started.ipynb

@@ -0,0 +1,197 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bda7b5add96e4d97",
+   "metadata": {},
+   "source": [
+    "# Getting Started with Trismik's Adaptive Testing\n",
+    "\n",
+    "This notebook demonstrates how to run Trismik's adaptive evaluations using Scorebook.\n",
+    "\n",
+    "## What is Adaptive Testing?\n",
+    "\n",
+    "Trismik’s adaptive testing service leverages item response theory (IRT), a psychometric framework, to evaluate large language models. Using computerized adaptive testing (CAT) it dynamically selects the most informative items, enabling faster, more cost-efficient model evaluations with fewer items required.\n",
+    "\n",
+    "## Setup\n",
+    "\n",
+    "### Install Scorebook\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "286eca2349c6ddc6",
+   "metadata": {},
+   "source": "!pip install scorebook",
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d2fa56528e14c46d",
+   "metadata": {},
+   "source": [
+    "\n",
+    "\n",
+    "### Generate a Trismik API Key\n",
+    "\n",
+    "To run an adaptive evaluation, a Trismik API key is required. You can [sign up](https://dashboard.trismik.com/signup) for a free Trismik account and generate an API key.\n",
+    "\n",
+    "**How to generate an API key from the Trismik dashboard**:\n",
+    "1. click on your initials in the top-right corner of the screen.\n",
+    "2. click on \"API Keys\" in the drop-down menu.\n",
+    "3. click \"Create API Key\" to create a new API key.\n",
+    "\n",
+    "### Set Trismik API Key"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "5ed8a62ac56560e9",
+   "metadata": {},
+   "source": [
+    "# Set your API key here and run this cell to login\n",
+    "TRISMIK_API_KEY = \"your-trismik-api-key-here\""
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "10d842e5b99e95bd",
+   "metadata": {},
+   "source": [
+    "### Login with Trismik API Key"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "initial_id",
+   "metadata": {},
+   "source": [
+    "import scorebook\n",
+    "\n",
+    "scorebook.login(TRISMIK_API_KEY)\n",
+    "print(\"✓ Logged in to Trismik\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eedbc41adba92248",
+   "metadata": {},
+   "source": [
+    "### Create a Project\n",
+    "\n",
+    "When running an adaptive evaluation, your evaluation results are stored under a project on the Trismik dashboard. Projects can be created from the dashboard's interface, or programmatically via Scorebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "bd7271800bf91478",
+   "metadata": {},
+   "source": [
+    "from scorebook import create_project\n",
+    "\n",
+    "# Create a project\n",
+    "project = create_project(\n",
+    "    name = \"Quick-Start Guides\",\n",
+    "    description = \"A project created for Trismik's quick-start guides.\"\n",
+    ")\n",
+    "\n",
+    "print(\"✓ Project created\")\n",
+    "print(f\"Project ID: {project.id}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e30ea2a2674e6005",
+   "metadata": {},
+   "source": [
+    "## Run an Adaptive Evaluation\n",
+    "\n",
+    "For this quick-start guide, we will use a mock model, that replicates the responses generated by an Amazon LLM (Nova-Pro). `mock_llm` is an inference function which accepts a list of model inputs and returns a list of model outputs for scoring. In this example, we use model responses that we pre-computed to showcase how adaptive evaluations work."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "d9f96d063e08c4fd",
+   "metadata": {},
+   "source": [
+    "from scorebook.utils.mock_llm import mock_llm\n",
+    "\n",
+    "# Run adaptive evaluation\n",
+    "results = scorebook.evaluate(\n",
+    "    inference = mock_llm,\n",
+    "    datasets = \"trismik/MMLUPro:adaptive\",\n",
+    "    experiment_id = \"Getting-Started\",\n",
+    "    project_id = project.id,\n",
+    ")\n",
+    "\n",
+    "# Print the adaptive evaluation results\n",
+    "print(\"✓ Adaptive evaluation complete!\")\n",
+    "print(\"Results: \", results[0][\"score\"])\n",
+    "\n",
+    "print(f\"You can view your results here: https://dashboard.trismik.com/projects/{project.id}\")"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "id": "63d3f34a76e90ac0",
+   "metadata": {},
+   "source": [
+    "### Adaptive Evaluation Results\n",
+    "\n",
+    "The metrics generated by an adaptive evaluation are:\n",
+    "\n",
+    "- Theta (θ): The primary score measuring model ability on the dataset, a higher value represents better performance.\n",
+    "- Standard Error: The theta score is a proxy for the underlying metric, and the standard error is the uncertainty in the theta estimate.\n",
+    "\n",
+    "You can find more information about adaptive testing [here](https://docs.trismik.com/adaptiveTesting/adaptive-testing-introduction/)!\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "**More Quick-Start Guides**:\n",
+    "\n",
+    "1. [HuggingFace Adaptive Evaluation](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/adaptive_evaluations/adaptive_evaluation_qwen_demo.ipynb): For a demo showcasing how Scorebook's adaptive evaluations can be used for local HF-based models. \n",
+    "2. [API-based Adaptive Evaluation](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/adaptive_evaluations/adaptive_evaluation_openai_demo.ipynb): For a demo showcasing how Scorebook's adaptive evaluations can be used for API-based models, using the OpenAI API.\n",
+    "3. [HuggingFace Full Dataset Evaluation](https://colab.research.google.com/github/trismik/scorebook/blob/main/tutorials/quickstarts/classical_evaluations/classical_evaluation_demo.ipynb): For a demo showcasing how Scorebook can be used for full dataset evaluations with familiar metrics, like `Accuracy`.\n",
+    "\n",
+    "**More details on Adaptive Testing and Scorebook**:\n",
+    "\n",
+    "- [Introduction to Adaptive Testing](https://docs.trismik.com/adaptiveTesting/adaptive-testing-introduction/): a quick introduction to Adaptive Testing.\n",
+    "- [Dataset Page](https://dashboard.trismik.com/datasets): Trismik's full set of currently adaptive datasets from the Trismik dashboard.\n",
+    "- [Scorebook Docs](https://docs.trismik.com/scorebook/introduction-to-scorebook/): Scorebook's full documentation.\n",
+    "- [Scorebook Repository](https://github.com/trismik/scorebook): Scorebook is an open-source library, view the code and more examples."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

tutorials/utils/__init__.py

@@ -0,0 +1,35 @@
+"""
+Helper utilities for Scorebook examples.
+
+This module provides common helper functions used across multiple Scorebook examples
+for setup, output handling, and argument parsing.
+"""
+
+# Argument parsing utilities
+from .args_parser import (
+    add_model_selection_arg,
+    create_parser,
+    parse_args_with_config,
+    setup_batch_model_parser,
+    setup_openai_model_parser,
+)
+
+# Output utilities
+from .output import save_results_to_json
+
+# Setup utilities
+from .setup import setup_logging, setup_output_directory
+
+__all__ = [
+    # Setup
+    "setup_logging",
+    "setup_output_directory",
+    # Output
+    "save_results_to_json",
+    # Argument parsing
+    "create_parser",
+    "add_model_selection_arg",
+    "setup_openai_model_parser",
+    "setup_batch_model_parser",
+    "parse_args_with_config",
+]

tutorials/utils/args_parser.py

@@ -0,0 +1,132 @@
+"""
+Generic argument parsing utilities for Scorebook examples.
+
+This module provides reusable argument parsing functions that can be used
+across multiple Scorebook examples for consistent command-line interfaces.
+"""
+
+import argparse
+from typing import Any, Dict, List, Optional
+
+
+def create_parser(description: str) -> argparse.ArgumentParser:
+    """Create a basic argument parser with a description.
+
+    Args:
+        description: Description for the argument parser
+
+    Returns:
+        Configured ArgumentParser instance
+    """
+    return argparse.ArgumentParser(description=description)
+
+
+def add_model_selection_arg(
+    parser: argparse.ArgumentParser,
+    default: str = "gpt-4o-mini",
+    help_text: Optional[str] = None,
+    supported_models: Optional[List[str]] = None,
+) -> argparse.ArgumentParser:
+    """Add model selection argument to parser.
+
+    Args:
+        parser: ArgumentParser to add the argument to
+        default: Default model name
+        help_text: Custom help text for the argument
+        supported_models: List of supported models for validation
+
+    Returns:
+        The modified parser
+    """
+    if help_text is None:
+        help_text = f"OpenAI model to use for inference (default: {default})"
+        if supported_models:
+            help_text += f". Supported models: {', '.join(supported_models)}"
+
+    parser.add_argument(
+        "--model",
+        type=str,
+        default=default,
+        help=help_text,
+    )
+    return parser
+
+
+def setup_openai_model_parser(
+    description: str = "Select OpenAI model for evaluation.",
+    default: str = "gpt-4o-mini",
+    supported_models: Optional[List[str]] = None,
+) -> str:
+    """Set up and parse OpenAI model selection arguments.
+
+    Args:
+        description: Description for the argument parser
+        default: Default model name
+        supported_models: List of supported models for help text
+
+    Returns:
+        Selected model name
+    """
+    parser = create_parser(description)
+    add_model_selection_arg(parser, default=default, supported_models=supported_models)
+    args = parser.parse_args()
+    return str(args.model)
+
+
+def setup_batch_model_parser(
+    description: str = "Select OpenAI model for batch evaluation.", default: str = "gpt-4o-mini"
+) -> str:
+    """Set up and parse OpenAI model selection arguments for batch inference.
+
+    Args:
+        description: Description for the argument parser
+        default: Default model name
+
+    Returns:
+        Selected model name
+    """
+    supported_models = ["gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "gpt-3.5-turbo"]
+    help_text = (
+        f"OpenAI model to use for batch inference. "
+        f"Note: Only select models support the Batch API. "
+        f"Supported models include: {', '.join(supported_models)}. "
+        f"Default: {default}"
+    )
+
+    parser = create_parser(description)
+    add_model_selection_arg(parser, default=default, help_text=help_text)
+    args = parser.parse_args()
+    return str(args.model)
+
+
+def parse_args_with_config(config: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:
+    """Parse arguments using a configuration dictionary.
+
+    Args:
+        config: Dictionary defining arguments to add. Format:
+            {
+                "arg_name": {
+                    "type": str,
+                    "default": "default_value",
+                    "help": "Help text",
+                    "required": False  # optional
+                }
+            }
+
+    Returns:
+        Dictionary of parsed argument values
+    """
+    parser = argparse.ArgumentParser()
+
+    for arg_name, arg_config in config.items():
+        kwargs = {"type": arg_config.get("type", str), "help": arg_config.get("help", "")}
+
+        if "default" in arg_config:
+            kwargs["default"] = arg_config["default"]
+        if "required" in arg_config:
+            kwargs["required"] = arg_config["required"]
+
+        parser.add_argument(f"--{arg_name.replace('_', '-')}", **kwargs)
+
+    args = parser.parse_args()
+    return vars(args)

tutorials/utils/output.py

@@ -0,0 +1,23 @@
+"""
+Utility functions for saving Scorebook evaluation results.
+
+This module provides common helper functions used across multiple Scorebook examples
+for saving evaluation results to files.
+"""
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+def save_results_to_json(results: Any, output_dir: Path, filename: str) -> None:
+    """Save evaluation results to a JSON file.
+
+    Args:
+        results: The evaluation results to save
+        output_dir: Directory to save the file in
+        filename: Name of the output file (should include .json extension)
+    """
+    output_path = output_dir / filename
+    with open(output_path, "w") as output_file:
+        json.dump(results, output_file, indent=4)

tutorials/utils/setup.py

@@ -0,0 +1,98 @@
+"""
+Utility functions for setting up Scorebook examples.
+
+This module provides common helper functions used across multiple Scorebook examples
+for output directory setup and logging configuration.
+"""
+
+import logging
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+
+def setup_output_directory() -> Path:
+    """Parse command line arguments and setup output directory."""
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Run evaluation and save results.")
+    parser.add_argument(
+        "--output-dir",
+        type=str,
+        default=str(Path.cwd() / "examples/example_results"),
+        help=(
+            "Directory to save evaluation outputs (CSV and JSON). "
+            "Defaults to ./examples/example_results in the current working directory."
+        ),
+    )
+    args = parser.parse_args()
+    output_dir = Path(args.output_dir)
+    output_dir.mkdir(parents=True, exist_ok=True)
+    print(f"Saving results to {output_dir}")
+    return output_dir
+
+
+def setup_logging(
+    log_dir: str = "logs",
+    experiment_id: Optional[str] = None,
+    base_dir: Optional[Path] = None,
+) -> Path:
+    """Configure logging for evaluation runs.
+
+    Args:
+        log_dir: Name of the log directory (default: "logs")
+        experiment_id: Optional identifier for the experiment
+        base_dir: Base directory where log_dir should be created.
+                  If None, uses current working directory.
+    """
+    if base_dir is None:
+        base_dir = Path.cwd()
+
+    log_dir_path: Path = base_dir / log_dir
+    log_dir_path.mkdir(exist_ok=True, parents=True)
+
+    timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
+    if experiment_id:
+        log_file = log_dir_path / f"evaluation_{experiment_id}_{timestamp}.log"
+    else:
+        log_file = log_dir_path / f"evaluation_{timestamp}.log"
+
+    # Create file handler for all logs (same as before)
+    file_handler = logging.FileHandler(log_file)
+    file_handler.setLevel(logging.DEBUG)
+    file_handler.setFormatter(
+        logging.Formatter("%(asctime)s - %(levelname)s - %(name)s - %(message)s")
+    )
+
+    # Create console handler for warnings and errors only
+    console_handler = logging.StreamHandler()
+    console_handler.setLevel(logging.WARNING)
+    console_handler.setFormatter(logging.Formatter("%(levelname)s - %(name)s - %(message)s"))
+
+    # Configure root logger with both handlers
+    logging.basicConfig(
+        level=logging.INFO,
+        handlers=[file_handler, console_handler],
+        force=True,
+    )
+
+    # Set scorebook loggers to DEBUG level to capture all scorebook logs
+    scorebook_logger = logging.getLogger("scorebook")
+    scorebook_logger.setLevel(logging.DEBUG)
+
+    # Ensure trismik_services logs are captured at DEBUG level
+    trismik_services_logger = logging.getLogger("scorebook.trismik_services")
+    trismik_services_logger.setLevel(logging.DEBUG)
+
+    # Ensure evaluate logs are captured at DEBUG level
+    evaluate_logger = logging.getLogger("scorebook.evaluate._sync.evaluate")
+    evaluate_logger.setLevel(logging.DEBUG)
+    evaluate_logger = logging.getLogger("scorebook.evaluate._async.evaluate_async")
+    evaluate_logger.setLevel(logging.DEBUG)
+
+    # Exclude OpenAI inference logs to reduce noise
+    openai_logger = logging.getLogger("scorebook.inference.openai")
+    openai_logger.setLevel(logging.WARNING)  # Only log warnings and errors
+
+    print(f"Logging to {log_file}")
+    return log_file

scorebook/metrics/metric_registry.py

@@ -1,105 +0,0 @@
-"""
-Registry module for evaluation metrics.
-
-This module maintains a centralized registry of available evaluation metrics
-that can be used to assess model performance. It provides a single access point
-to retrieve all implemented metric classes.
-"""
-
-from typing import Any, Callable, Dict, List, Type, Union
-
-from scorebook.metrics.metric_base import MetricBase
-
-
-class MetricRegistry:
-    """A registry for evaluation metrics.
-
-    This class provides a central registry for all evaluation metrics in the system.
-    It allows metrics to be registered with unique names and retrieved either by
-    name or by class. The registry ensures that metrics are properly initialized
-    and accessible throughout the application.
-
-    The registry supports:
-    - Registering new metric classes with optional custom names
-    - Retrieving metric instances by name or class
-    - Listing all available metrics
-
-    Usage:
-        @MetricRegistry.register("custom_name")
-        class MyMetric(MetricBase):
-            ...
-
-        # Get by name
-        metric = MetricRegistry.get("custom_name")
-
-        # Get by class
-        metric = MetricRegistry.get(MyMetric)
-
-        # List available metrics
-        metrics = MetricRegistry.list_metrics()
-    """
-
-    _registry: Dict[str, Type[MetricBase]] = {}
-
-    @classmethod
-    def register(cls) -> Callable[[Type[MetricBase]], Type[MetricBase]]:
-        """
-        Register a metric class in the registry.
-
-        Returns:
-            A decorator that registers the class and returns it.
-
-        Raises:
-            ValueError: If a metric with the given name is already registered.
-        """
-
-        def decorator(metric_cls: Type[MetricBase]) -> Type[MetricBase]:
-
-            key = metric_cls.__name__.lower()
-            if key in cls._registry:
-                raise ValueError(f"Metric '{key}' is already registered")
-            cls._registry[key] = metric_cls
-            return metric_cls
-
-        return decorator
-
-    @classmethod
-    def get(cls, name_or_class: Union[str, Type[MetricBase]], **kwargs: Any) -> MetricBase:
-        """
-        Get an instance of a registered metric by name or class.
-
-        Args:
-            name_or_class: The metric name (string) or class (subclass of BaseMetric).
-            **kwargs: Additional arguments to pass to the metric's constructor.
-
-        Returns:
-            An instance of the requested metric.
-
-        Raises:
-            ValueError: If the metric name is not registered.
-        """
-        # If input is a class that's a subclass of BaseMetric, instantiate it directly
-        if isinstance(name_or_class, type) and issubclass(name_or_class, MetricBase):
-            return name_or_class(**kwargs)
-
-        # If input is a string, look up the class in the registry
-        if isinstance(name_or_class, str):
-            key = name_or_class.lower()
-            if key not in cls._registry:
-                raise ValueError(f"Metric '{name_or_class}' not registered.")
-            return cls._registry[key](**kwargs)
-
-        raise ValueError(
-            f"Invalid metric type: {type(name_or_class)}."
-            f"Must be string name or BaseMetric subclass"
-        )
-
-    @classmethod
-    def list_metrics(cls) -> List[str]:
-        """
-        List all registered metrics.
-
-        Returns:
-            A list of metric names.
-        """
-        return list(cls._registry.keys())

scorebook/trismik/__init__.py

@@ -1,10 +0,0 @@
-"""Trismik authentication and API integration.
-
-Note: Trismik evaluation functionality has been moved to scorebook.evaluate module.
-This module now only provides authentication functions.
-"""
-
-# Import shared credential functions
-from .credentials import get_stored_token, get_token, login, logout, whoami
-
-__all__ = ["login", "logout", "whoami", "get_stored_token", "get_token"]