experiment-configuration-agent 0.1.0__tar.gz

experiment_configuration_agent-0.1.0/PKG-INFO

@@ -0,0 +1,117 @@
+Metadata-Version: 2.4
+Name: experiment-configuration-agent
+Version: 0.1.0
+Summary: Add your description here
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic-settings
+Requires-Dist: sfn-blueprint>=0.6.16
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-mock; extra == "dev"
+
+# Experiment Configuration Agent for AutoGluon
+
+This agent uses a Large Language Model to recommend optimal configurations for AutoGluon's `TabularPredictor` based on your machine learning problem context. By providing details about your domain, use case, and dataset, the agent will generate a set of `TabularPredictor` parameters designed to optimize for performance and efficiency.
+
+## Features
+
+-   **Intelligent Configuration:** Leverages LLMs to recommend `eval_metric`, `presets`, `time_limit`, and ensembling parameters.
+-   **Context-Aware:** Considers the business domain, specific use case, ML methodology (e.g., classification, regression), and dataset characteristics.
+-   **Flexible Backend:** Powered by `sfn-blueprint`, allowing for a configurable LLM backend.
+-   **Multiple Scenarios:** Provides recommendations for different optimization goals, such as maximizing accuracy, balancing performance and speed, or fast prototyping.
+
+## Installation
+
+This project uses `uv` for dependency management and requires Python 3.10 or higher.
+
+1.  **Clone the repository:**
+    ```bash
+    git clone <repository-url>
+    cd experiment-configuration-agent
+    ```
+
+2.  **Set up the environment and install dependencies:**
+    It is recommended to use a virtual environment. `uv` can create one for you.
+    ```bash
+    # Install uv if you don't have it
+    pip install uv
+
+    # Create a virtual environment and install dependencies
+    uv sync
+    ```
+
+## Usage
+
+To get a configuration recommendation, instantiate the `AutoGluonConfigAgent` and pass a dictionary containing the problem context.
+
+1.  **Create a `.env` file** in the project root to configure the LLM provider. See the [Configuration](#configuration) section for more details.
+
+    ```
+    PROVIDER="openai"
+    MODEL="gpt-4-turbo"
+    # Add your API key, e.g., OPENAI_API_KEY="sk-..."
+    ```
+
+2.  **Create your Python script:**
+
+    ```python
+    from experiment_configuration_agent.agent import AutoGluonConfigAgent
+
+    # 1. Define the problem context
+    task_data = {
+        "domain": {
+            "name": "Manufacturing",
+            "description": "An automotive parts manufacturing facility with multiple production lines."
+        },
+        "use_case": {
+            "name": "Predictive Maintenance",
+            "description": "Detect unusual temporal patterns in sensor data to predict equipment failure and prevent breakdowns."
+        },
+        "methodology": "binary_classification",
+        "dataset_insights": {
+            "num_samples": 5000,
+            "num_features": 10,
+            "target": {
+                "name": "failure_flag",
+                "imbalance_ratio": 0.05 # Highly imbalanced
+            },
+            "feature_summary": {
+                "sensor_A": {"min": 0.1, "max": 100.5, "dtype": "float"},
+                "production_line_id": {"unique_count": 3, "dtype": "category"}
+            }
+        }
+    }
+
+    # 2. Initialize the agent
+    agent = AutoGluonConfigAgent()
+
+    # 3. Get the configuration recommendation
+    result = agent(task_data)
+
+    # 4. Print the result
+    print("Recommended AutoGluon Configuration:")
+    print(result.get("configuration"))
+    print("\nCost Summary:")
+    print(result.get("cost_summary"))
+
+    ```
+
+## Configuration
+
+The agent is configured via environment variables, which can be placed in a `.env` file in the project root. The primary configurations are inherited from the `GluonConfig` class.
+
+-   `PROVIDER`: The LLM provider to use (e.g., `"openai"`, `"anthropic"`).
+-   `MODEL`: The specific model to use (e.g., `"gpt-4-turbo"`, `"claude-3-opus-20240229"`).
+-   `TEMPERATURE`: The model's temperature setting (e.g., `0.3`).
+-   `MAX_TOKENS`: The maximum number of tokens for the response (e.g., `4000`).
+
+You will also need to set the API key for your chosen provider, for example `OPENAI_API_KEY="your-key-here"`.
+
+## Testing
+
+This project uses `pytest`. To run the test suite, execute the following command from the project root:
+
+```bash
+pytest
+```

experiment_configuration_agent-0.1.0/README.md

@@ -0,0 +1,105 @@
+# Experiment Configuration Agent for AutoGluon
+
+This agent uses a Large Language Model to recommend optimal configurations for AutoGluon's `TabularPredictor` based on your machine learning problem context. By providing details about your domain, use case, and dataset, the agent will generate a set of `TabularPredictor` parameters designed to optimize for performance and efficiency.
+
+## Features
+
+-   **Intelligent Configuration:** Leverages LLMs to recommend `eval_metric`, `presets`, `time_limit`, and ensembling parameters.
+-   **Context-Aware:** Considers the business domain, specific use case, ML methodology (e.g., classification, regression), and dataset characteristics.
+-   **Flexible Backend:** Powered by `sfn-blueprint`, allowing for a configurable LLM backend.
+-   **Multiple Scenarios:** Provides recommendations for different optimization goals, such as maximizing accuracy, balancing performance and speed, or fast prototyping.
+
+## Installation
+
+This project uses `uv` for dependency management and requires Python 3.10 or higher.
+
+1.  **Clone the repository:**
+    ```bash
+    git clone <repository-url>
+    cd experiment-configuration-agent
+    ```
+
+2.  **Set up the environment and install dependencies:**
+    It is recommended to use a virtual environment. `uv` can create one for you.
+    ```bash
+    # Install uv if you don't have it
+    pip install uv
+
+    # Create a virtual environment and install dependencies
+    uv sync
+    ```
+
+## Usage
+
+To get a configuration recommendation, instantiate the `AutoGluonConfigAgent` and pass a dictionary containing the problem context.
+
+1.  **Create a `.env` file** in the project root to configure the LLM provider. See the [Configuration](#configuration) section for more details.
+
+    ```
+    PROVIDER="openai"
+    MODEL="gpt-4-turbo"
+    # Add your API key, e.g., OPENAI_API_KEY="sk-..."
+    ```
+
+2.  **Create your Python script:**
+
+    ```python
+    from experiment_configuration_agent.agent import AutoGluonConfigAgent
+
+    # 1. Define the problem context
+    task_data = {
+        "domain": {
+            "name": "Manufacturing",
+            "description": "An automotive parts manufacturing facility with multiple production lines."
+        },
+        "use_case": {
+            "name": "Predictive Maintenance",
+            "description": "Detect unusual temporal patterns in sensor data to predict equipment failure and prevent breakdowns."
+        },
+        "methodology": "binary_classification",
+        "dataset_insights": {
+            "num_samples": 5000,
+            "num_features": 10,
+            "target": {
+                "name": "failure_flag",
+                "imbalance_ratio": 0.05 # Highly imbalanced
+            },
+            "feature_summary": {
+                "sensor_A": {"min": 0.1, "max": 100.5, "dtype": "float"},
+                "production_line_id": {"unique_count": 3, "dtype": "category"}
+            }
+        }
+    }
+
+    # 2. Initialize the agent
+    agent = AutoGluonConfigAgent()
+
+    # 3. Get the configuration recommendation
+    result = agent(task_data)
+
+    # 4. Print the result
+    print("Recommended AutoGluon Configuration:")
+    print(result.get("configuration"))
+    print("\nCost Summary:")
+    print(result.get("cost_summary"))
+
+    ```
+
+## Configuration
+
+The agent is configured via environment variables, which can be placed in a `.env` file in the project root. The primary configurations are inherited from the `GluonConfig` class.
+
+-   `PROVIDER`: The LLM provider to use (e.g., `"openai"`, `"anthropic"`).
+-   `MODEL`: The specific model to use (e.g., `"gpt-4-turbo"`, `"claude-3-opus-20240229"`).
+-   `TEMPERATURE`: The model's temperature setting (e.g., `0.3`).
+-   `MAX_TOKENS`: The maximum number of tokens for the response (e.g., `4000`).
+
+You will also need to set the API key for your chosen provider, for example `OPENAI_API_KEY="your-key-here"`.
+
+## Testing
+
+This project uses `pytest`. To run the test suite, execute the following command from the project root:
+
+```bash
+pytest
+```

experiment_configuration_agent-0.1.0/pyproject.toml

@@ -0,0 +1,27 @@
+[project]
+name = "experiment-configuration-agent"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "pydantic-settings",
+    "sfn-blueprint>=0.6.16",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest",
+    "pytest-mock",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+pythonpath = [
+  "src"
+]
+filterwarnings = [
+    "ignore::DeprecationWarning",
+]

experiment_configuration_agent-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

experiment_configuration_agent-0.1.0/src/experiment_config_agent/agent.py

@@ -0,0 +1,57 @@
+
+from sfn_blueprint import BaseLangChainAgent
+from .config import GluonConfig
+from .constants import format_autogluon_config_prompt
+from .models import AutoGluonConfig
+from typing import Dict, Any, Tuple, Optional
+import logging
+
+
+class AutoGluonConfigAgent(BaseLangChainAgent):
+    def __init__(self, config: Optional[GluonConfig] = None):
+        super().__init__(config or GluonConfig())
+
+    def configure_training(
+        self, 
+        domain: Dict[str, str], 
+        use_case: Dict[str, str], 
+        methodology: str, 
+        dataset_insights: Dict[str, Any]
+    ) :
+        """
+        Generates an AutoGluon configuration based on domain context and data insights.
+        """
+
+        system_prompt, user_prompt = format_autogluon_config_prompt(
+            domain=domain, 
+            use_case=use_case, 
+            methodology=methodology, 
+            dataset_insights=dataset_insights
+        )
+
+        response, cost_summary = self.route_with_langchain(
+            system_prompt, 
+            user_prompt, 
+            AutoGluonConfig
+        )
+
+        return response, cost_summary
+
+    def execute_task(self, task_data: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Wrapper to execute via standard task dictionary interface.
+        """
+
+        config_recommendation, cost = self.configure_training(
+            domain=task_data["domain"],
+            use_case=task_data["use_case"],
+            methodology=task_data["methodology"],
+            dataset_insights=task_data["dataset_insights"]
+        )
+
+        return {
+            "configuration": config_recommendation.model_dump(),
+            "cost_summary": cost
+        }
+    def __call__(self, task_data: Dict[str, Any]) -> Dict[str, Any]:
+        return self.execute_task(task_data)

experiment_configuration_agent-0.1.0/src/experiment_config_agent/config.py

@@ -0,0 +1,20 @@
+from pydantic import Field
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+
+
+class GluonConfig(BaseSettings):
+    model_config = SettingsConfigDict(
+        env_file='.env',
+        env_file_encoding='utf-8',
+        case_sensitive=False,
+        extra='ignore'
+    )
+
+    provider: str = Field(default="openai", description="AI provider to use")
+    model: str = Field(default="gpt-5-mini", description="AI model to use")
+    temperature: float = Field(default=0.3, ge=0.0, le=0.5, description="AI model temperature")
+    max_tokens: int = Field(default=4000, ge=0, le=8000, description="Maximum tokens for AI response")
+
+

experiment_configuration_agent-0.1.0/src/experiment_config_agent/constants.py

@@ -0,0 +1,235 @@
+AUTOGLUON_CONFIG_SYSTEM_PROMPT = """You are an expert AutoGluon configuration advisor specializing in optimizing TabularPredictor settings for various machine learning problems.
+
+Your role is to analyze the provided information about:
+1. The domain and business context
+2. The specific use case and problem type
+3. The methodology (classification type or regression)
+4. Dataset characteristics from the modeling-ready insights
+
+Based on this analysis, you must recommend optimal AutoGluon TabularPredictor configuration parameters that balance:
+- Predictive accuracy and model quality
+- Training time and computational resources
+- Inference speed requirements
+- Model complexity and interpretability needs
+- Domain-specific requirements
+
+Key AutoGluon Concepts:
+======================
+
+EVALUATION METRICS:
+- For binary classification: f1, precision, recall, accuracy, roc_auc, balanced_accuracy, log_loss, mcc
+- For multiclass classification: accuracy, balanced_accuracy, log_loss, f1_macro, f1_micro, f1_weighted
+- For regression: rmse, mae, r2, mse, mape
+- Choose metrics based on business requirements (e.g., f1 for imbalanced classes, accuracy for balanced classes)
+
+PRESETS (Quality vs Speed Tradeoff):
+- "best_quality": Maximum accuracy with auto_stack=True, uses bagging/stacking (slowest, most accurate)
+- "high_quality": Strong accuracy with fast inference, moderate complexity
+- "good_quality": Good accuracy with very fast inference (recommended default)
+- "medium_quality": Fast training time, ideal for prototyping
+- "optimize_for_deployment": Minimal memory/compute footprint
+- "interpretable": Simple, explainable models (e.g., linear models, shallow trees)
+
+BAGGING & STACKING (Ensemble Methods):
+- num_bag_folds: 0=no bagging, 5-10=k-fold bagging (improves accuracy, increases training time)
+- num_bag_sets: Number of times to repeat k-fold bagging (1-3, only if num_bag_folds>0)
+- num_stack_levels: 0=no stacking, 1=single-level stacking, 2+=multi-level stacking
+- auto_stack: True=AutoGluon automatically determines optimal stacking/bagging (recommended for best quality)
+- refit_full: True=retrain on full dataset after validation (essential when bagging is used)
+
+TIME CONSTRAINTS:
+- time_limit: Total seconds for training (60-3600+ depending on dataset size)
+- Larger datasets need more time, more complex models need more time
+- Consider: small datasets (<10K rows): 60-300s, medium (10K-100K): 300-1800s, large (>100K): 1800-7200s
+
+INFERENCE SPEED:
+- infer_limit: Max seconds per row prediction (e.g., 0.001=1ms, 0.00005=0.05ms)
+- infer_limit_batch_size: Batch size for speed calculation (1=online, 1000+=batch inference)
+- Only specify if real-time inference speed is critical
+
+DECISION THRESHOLD CALIBRATION (Binary Classification):
+- "auto": Calibrate for metrics like f1, balanced_accuracy (recommended)
+- True: Always calibrate
+- False: Never calibrate
+- Significantly improves f1, balanced_accuracy, and recall-related metrics
+
+Domain-Specific Considerations:
+==============================
+
+MANUFACTURING/QUALITY CONTROL:
+- High precision to avoid false positives (unnecessary inspections)
+- Use precision, f1 metrics
+- Consider interpretable models for regulatory compliance
+- Fast inference for real-time monitoring
+
+HEALTHCARE/MEDICAL:
+- High recall to avoid missing critical cases
+- Use recall, balanced_accuracy, f1 metrics
+- Interpretability is often critical
+- May need decision threshold calibration
+
+FRAUD DETECTION:
+- Highly imbalanced classes
+- Use f1, balanced_accuracy, roc_auc, precision-recall metrics
+- Fast inference for real-time detection
+- Consider calibrate_decision_threshold=True
+
+PREDICTIVE MAINTENANCE:
+- Balance precision and recall
+- Consider temporal patterns
+- Fast inference for real-time monitoring
+- Good interpretability for actionable insights
+
+CUSTOMER CHURN/RETENTION:
+- Focus on identifying at-risk customers (recall)
+- Use f1, balanced_accuracy, roc_auc
+- Interpretability helps with intervention strategies
+
+FINANCIAL/CREDIT SCORING:
+- Regulatory requirements for interpretability
+- Balance precision and recall
+- Consider "interpretable" preset or simple models
+
+Dataset Characteristics Impact:
+==============================
+
+SMALL DATASETS (<10K rows):
+- Benefit greatly from bagging (num_bag_folds=5-8)
+- May overfit with too many models
+- Shorter time_limit (60-300s)
+- Consider cross-validation
+
+IMBALANCED CLASSES:
+- Use f1, balanced_accuracy, roc_auc metrics (NOT accuracy)
+- Consider calibrate_decision_threshold=True
+- May benefit from custom hyperparameters
+
+HIGH-DIMENSIONAL DATA (many features):
+- May need longer time_limit
+- Feature importance analysis post-training
+- Consider preset="best_quality" for feature interactions
+
+TEMPORAL/TIME-SERIES PATTERNS:
+- Be cautious with shuffling (AutoGluon does stratified splits)
+- May need time-based validation splits
+- Consider specialized time-series features
+
+Be thoughtful, analytical, and provide production-ready recommendations."""
+
+
+
+def format_autogluon_config_prompt(
+    domain: dict,
+    use_case: str,
+    methodology: str,
+    dataset_insights: dict
+) -> tuple[str, str]:
+    """
+    Format the system and user prompts for AutoGluon configuration recommendation.
+    
+    Args:
+        domain_name: Name of the business domain
+        domain_description: Detailed description of the domain context
+        use_case: Description of the specific use case and problem
+        methodology: Type of ML problem (binary_classification, multiclass_classification, regression)
+        dataset_insights: Dictionary containing feature and target information
+        
+    Returns:
+        Tuple of (system_prompt, user_prompt)
+    """
+    
+    # feature_columns = dataset_insights.get('feature_columns', {})
+    # target_info = dataset_insights.get('target', {})
+    
+    # feature_summary = []
+    # for col_name, col_info in feature_columns.items():
+    #     feature_summary.append(
+    #         f"  - {col_name}: "
+    #         f"type={col_info.get('dtype', 'unknown')}, "
+    #         f"missing={col_info.get('missing_pct', 0):.1f}%, "
+    #         f"unique={col_info.get('unique_count', 'N/A')}"
+    #     )
+    #     if 'min' in col_info and 'max' in col_info:
+    #         feature_summary.append(f"    range=[{col_info['min']}, {col_info['max']}]")
+    
+    # feature_text = "\n".join(feature_summary) if feature_summary else "No feature information provided"
+    
+    # # Format target information
+    # target_text = []
+    # if target_info:
+    #     target_text.append(f"Target Column: {target_info.get('name', 'unknown')}")
+    #     target_text.append(f"  Type: {target_info.get('dtype', 'unknown')}")
+        
+    #     if 'class_distribution' in target_info:
+    #         target_text.append("  Class Distribution:")
+    #         for cls, count in target_info['class_distribution'].items():
+    #             target_text.append(f"    - {cls}: {count}")
+        
+    #     if 'min' in target_info and 'max' in target_info:
+    #         target_text.append(f"  Range: [{target_info['min']}, {target_info['max']}]")
+            
+    #     if 'mean' in target_info:
+    #         target_text.append(f"  Mean: {target_info['mean']:.2f}")
+    
+    # target_summary = "\n".join(target_text) if target_text else "No target information provided"
+    
+    # # Get dataset size information
+    # num_samples = dataset_insights.get('num_samples', 'unknown')
+    # num_features = len(feature_columns) if feature_columns else 'unknown'
+
+    # DATASET INSIGHTS:
+    # ================
+    # Number of Samples: {num_samples}
+    # Number of Features: {num_features}
+
+    # Features:
+    # {feature_text}
+
+    # {target_summary}
+    # 8. hyperparameters: Hyperparameter preset ("default", "light", "very_light")
+    # 9. auto_stack: Whether to use automatic stacking (true/false)
+    # 10. infer_limit: Max inference time per row in seconds (or null)
+    # 11. infer_limit_batch_size: Batch size for inference speed (or null)
+    # 12. refit_full: Whether to retrain on full data (true/false)
+    # 13. calibrate_decision_threshold: Threshold calibration setting ("auto", true, false)
+
+    
+    user_prompt = f"""Please recommend optimal AutoGluon TabularPredictor configuration for the following scenario:
+
+DOMAIN INFORMATION:
+==================
+Domain: {domain}
+
+USE CASE:
+=========
+{use_case}
+
+METHODOLOGY:
+===========
+Problem Type: {methodology}
+
+DATASET INSIGHTS:
+================
+{dataset_insights}
+
+TASK:
+=====
+Based on the above information, recommend an optimal AutoGluon configuration that includes:
+
+1. eval_metric: The primary metric to optimize
+2. preset: Quality/speed tradeoff preset  
+3. additional_metrics: Other metrics to track (list)
+4. time_limit: Training time in seconds
+5. num_bag_folds: Number of k-fold bagging folds (0 for none, 5-10 for bagging)
+6. num_bag_sets: Number of bagging sets (1-3, only if bagging is used)
+7. num_stack_levels: Number of stacking levels 
+
+
+Consider multiple scenarios:
+- Scenario A: Maximum accuracy (accepting longer training time)
+- Scenario B: Balanced accuracy and speed (production-ready)
+- Scenario C: Fast training and inference (prototyping/deployment constrained)
+
+"""
+    
+    return AUTOGLUON_CONFIG_SYSTEM_PROMPT, user_prompt

experiment_configuration_agent-0.1.0/src/experiment_config_agent/models.py

@@ -0,0 +1,30 @@
+from pydantic import BaseModel, Field
+from typing import List, Literal, Optional
+
+
+class AutoGluonConfig(BaseModel):
+    eval_metric: str = Field(..., description="Primary metric to optimize. For binary classification: 'f1', 'precision', 'recall', 'accuracy', 'roc_auc', 'balanced_accuracy', 'log_loss'. For multiclass: 'accuracy', 'balanced_accuracy', 'log_loss', 'f1_macro', 'f1_micro', 'f1_weighted'. For regression: 'rmse', 'mae', 'r2', 'mse'")
+    
+    preset: Literal["best_quality", "high_quality", "good_quality", "medium_quality", "optimize_for_deployment", "interpretable"] = Field(..., description="Preset configurations that control model complexity and training strategy. 'best_quality' for maximum accuracy with stacking/bagging, 'high_quality' for strong accuracy with fast inference, 'good_quality' for good accuracy with very fast inference, 'medium_quality' for fast training, 'optimize_for_deployment' for minimal memory/compute, 'interpretable' for simple models")
+    
+    additional_metrics: List[str] = Field(..., description="List of additional metrics to track during training. Options: 'f1', 'precision', 'recall', 'accuracy', 'roc_auc', 'balanced_accuracy', 'log_loss', 'rmse', 'mae', 'r2', 'mse'")
+    
+    time_limit: int = Field(..., description="Time limit in seconds for training all models. Recommended: 60-3600 seconds depending on dataset size and complexity")
+    
+    num_bag_folds: int = Field(..., description="Number of folds for k-fold bagging. 0 = no bagging, 5-10 = good for improving accuracy with longer training time. Higher values reduce overfitting but increase training time and memory usage")
+    
+    num_bag_sets: int = Field(..., description="Number of bagging sets. Each set repeats k-fold bagging to further reduce variance. 1 = standard bagging, 2-3 = better accuracy but much longer training. Only relevant when num_bag_folds > 0")
+    
+    num_stack_levels: int = Field(..., description="Number of stacking levels. 0 = no stacking, 1 = one level of stacking (models trained on predictions of base models), 2+ = multi-level stacking. Higher values can improve accuracy but increase training time exponentially")
+    
+    # hyperparameters: Optional[str] = Field(default="default", description="Hyperparameters preset. 'default' = balanced set of models, 'light' = fast training with simpler models, 'very_light' = very fast training, 'toy' = minimal training for testing. Can also be a custom dict")
+    
+    # auto_stack: bool = Field(..., description="If True, automatically determines optimal stacking/bagging configuration. Recommended for best quality. Overrides num_bag_folds, num_bag_sets, num_stack_levels")
+    
+    # infer_limit: Optional[float] = Field(..., description="Maximum time in seconds to predict a single row. If specified, AutoGluon will only train models that meet this constraint. Example: 0.001 = 1ms per row, 0.00005 = 0.05ms per row (20000 rows/sec)")
+    
+    # infer_limit_batch_size: Optional[int] = Field(..., description="Batch size for inference speed calculation when infer_limit is specified. 1 = online inference (strict), 1000+ = batch inference (easier to satisfy). Must be specified if infer_limit is set")
+    
+    # refit_full: bool = Field(..., description="Whether to retrain models on full dataset after initial training. Significantly improves quality when bagging is used. Recommended: True if bagging is enabled")
+    
+    # calibrate_decision_threshold: Literal["auto", True, False] = Field(..., description="Whether to calibrate the decision threshold for binary classification. 'auto' = calibrate for metrics like f1, balanced_accuracy but not for accuracy, True = always calibrate, False = never calibrate")

experiment_configuration_agent-0.1.0/src/experiment_configuration_agent.egg-info/PKG-INFO

@@ -0,0 +1,117 @@
+Metadata-Version: 2.4
+Name: experiment-configuration-agent
+Version: 0.1.0
+Summary: Add your description here
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic-settings
+Requires-Dist: sfn-blueprint>=0.6.16
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-mock; extra == "dev"
+
+# Experiment Configuration Agent for AutoGluon
+
+This agent uses a Large Language Model to recommend optimal configurations for AutoGluon's `TabularPredictor` based on your machine learning problem context. By providing details about your domain, use case, and dataset, the agent will generate a set of `TabularPredictor` parameters designed to optimize for performance and efficiency.
+
+## Features
+
+-   **Intelligent Configuration:** Leverages LLMs to recommend `eval_metric`, `presets`, `time_limit`, and ensembling parameters.
+-   **Context-Aware:** Considers the business domain, specific use case, ML methodology (e.g., classification, regression), and dataset characteristics.
+-   **Flexible Backend:** Powered by `sfn-blueprint`, allowing for a configurable LLM backend.
+-   **Multiple Scenarios:** Provides recommendations for different optimization goals, such as maximizing accuracy, balancing performance and speed, or fast prototyping.
+
+## Installation
+
+This project uses `uv` for dependency management and requires Python 3.10 or higher.
+
+1.  **Clone the repository:**
+    ```bash
+    git clone <repository-url>
+    cd experiment-configuration-agent
+    ```
+
+2.  **Set up the environment and install dependencies:**
+    It is recommended to use a virtual environment. `uv` can create one for you.
+    ```bash
+    # Install uv if you don't have it
+    pip install uv
+
+    # Create a virtual environment and install dependencies
+    uv sync
+    ```
+
+## Usage
+
+To get a configuration recommendation, instantiate the `AutoGluonConfigAgent` and pass a dictionary containing the problem context.
+
+1.  **Create a `.env` file** in the project root to configure the LLM provider. See the [Configuration](#configuration) section for more details.
+
+    ```
+    PROVIDER="openai"
+    MODEL="gpt-4-turbo"
+    # Add your API key, e.g., OPENAI_API_KEY="sk-..."
+    ```
+
+2.  **Create your Python script:**
+
+    ```python
+    from experiment_configuration_agent.agent import AutoGluonConfigAgent
+
+    # 1. Define the problem context
+    task_data = {
+        "domain": {
+            "name": "Manufacturing",
+            "description": "An automotive parts manufacturing facility with multiple production lines."
+        },
+        "use_case": {
+            "name": "Predictive Maintenance",
+            "description": "Detect unusual temporal patterns in sensor data to predict equipment failure and prevent breakdowns."
+        },
+        "methodology": "binary_classification",
+        "dataset_insights": {
+            "num_samples": 5000,
+            "num_features": 10,
+            "target": {
+                "name": "failure_flag",
+                "imbalance_ratio": 0.05 # Highly imbalanced
+            },
+            "feature_summary": {
+                "sensor_A": {"min": 0.1, "max": 100.5, "dtype": "float"},
+                "production_line_id": {"unique_count": 3, "dtype": "category"}
+            }
+        }
+    }
+
+    # 2. Initialize the agent
+    agent = AutoGluonConfigAgent()
+
+    # 3. Get the configuration recommendation
+    result = agent(task_data)
+
+    # 4. Print the result
+    print("Recommended AutoGluon Configuration:")
+    print(result.get("configuration"))
+    print("\nCost Summary:")
+    print(result.get("cost_summary"))
+
+    ```
+
+## Configuration
+
+The agent is configured via environment variables, which can be placed in a `.env` file in the project root. The primary configurations are inherited from the `GluonConfig` class.
+
+-   `PROVIDER`: The LLM provider to use (e.g., `"openai"`, `"anthropic"`).
+-   `MODEL`: The specific model to use (e.g., `"gpt-4-turbo"`, `"claude-3-opus-20240229"`).
+-   `TEMPERATURE`: The model's temperature setting (e.g., `0.3`).
+-   `MAX_TOKENS`: The maximum number of tokens for the response (e.g., `4000`).
+
+You will also need to set the API key for your chosen provider, for example `OPENAI_API_KEY="your-key-here"`.
+
+## Testing
+
+This project uses `pytest`. To run the test suite, execute the following command from the project root:
+
+```bash
+pytest
+```

experiment_configuration_agent-0.1.0/src/experiment_configuration_agent.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+README.md
+pyproject.toml
+src/experiment_config_agent/__init__.py
+src/experiment_config_agent/agent.py
+src/experiment_config_agent/config.py
+src/experiment_config_agent/constants.py
+src/experiment_config_agent/models.py
+src/experiment_configuration_agent.egg-info/PKG-INFO
+src/experiment_configuration_agent.egg-info/SOURCES.txt
+src/experiment_configuration_agent.egg-info/dependency_links.txt
+src/experiment_configuration_agent.egg-info/requires.txt
+src/experiment_configuration_agent.egg-info/top_level.txt
+tests/test_agent.py

experiment_configuration_agent-0.1.0/src/experiment_configuration_agent.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

experiment_configuration_agent-0.1.0/src/experiment_configuration_agent.egg-info/requires.txt

@@ -0,0 +1,6 @@
+pydantic-settings
+sfn-blueprint>=0.6.16
+
+[dev]
+pytest
+pytest-mock

experiment_configuration_agent-0.1.0/src/experiment_configuration_agent.egg-info/top_level.txt

@@ -0,0 +1 @@
+experiment_config_agent

experiment_configuration_agent-0.1.0/tests/test_agent.py

@@ -0,0 +1,89 @@
+import pytest
+from unittest.mock import patch, MagicMock
+from experiment_config_agent.agent import AutoGluonConfigAgent
+from experiment_config_agent.models import AutoGluonConfig
+
+@pytest.fixture
+def agent():
+    """Fixture to create an AutoGluonConfigAgent instance."""
+    return AutoGluonConfigAgent()
+
+@pytest.fixture
+def task_data():
+    """Fixture to provide sample task data."""
+    return {
+        "domain": {"name": "Finance", "description": "Credit scoring"},
+        "use_case": {"name": "Loan Default Prediction", "description": "Predict if a customer will default on a loan"},
+        "methodology": "binary_classification",
+        "dataset_insights": {"samples": 10000, "features": 15}
+    }
+
+def test_configure_training(agent, task_data):
+    """Test the configure_training method."""
+    mock_response = AutoGluonConfig(
+        eval_metric='f1',
+        preset='high_quality',
+        additional_metrics=['roc_auc', 'accuracy'],
+        time_limit=3600,
+        num_bag_folds=5,
+        num_bag_sets=1,
+        num_stack_levels=1
+    )
+    mock_cost = {"total_cost": 0.05}
+
+    with patch.object(AutoGluonConfigAgent, 'route_with_langchain', return_value=(mock_response, mock_cost)) as mock_route:
+        response, cost_summary = agent.configure_training(
+            domain=task_data["domain"],
+            use_case=task_data["use_case"],
+            methodology=task_data["methodology"],
+            dataset_insights=task_data["dataset_insights"]
+        )
+
+        assert response == mock_response
+        assert cost_summary == mock_cost
+        mock_route.assert_called_once()
+        system_prompt, user_prompt, model = mock_route.call_args[0]
+        assert "You are an expert AutoGluon configuration advisor" in system_prompt
+        assert "DOMAIN INFORMATION" in user_prompt
+
+
+def test_execute_task(agent, task_data):
+    """Test the execute_task method."""
+    mock_config = AutoGluonConfig(
+        eval_metric='f1',
+        preset='good_quality',
+        additional_metrics=['accuracy'],
+        time_limit=600,
+        num_bag_folds=0,
+        num_bag_sets=0,
+        num_stack_levels=0
+    )
+    mock_cost = {"total_cost": 0.02}
+    
+    expected_config_dump = mock_config.model_dump()
+
+    with patch.object(AutoGluonConfigAgent, 'configure_training', return_value=(mock_config, mock_cost)) as mock_configure:
+        result = agent.execute_task(task_data)
+
+        mock_configure.assert_called_once_with(
+            domain=task_data["domain"],
+            use_case=task_data["use_case"],
+            methodology=task_data["methodology"],
+            dataset_insights=task_data["dataset_insights"]
+        )
+        
+        assert "configuration" in result
+        assert "cost_summary" in result
+        assert result["configuration"] == expected_config_dump
+        assert result["cost_summary"] == mock_cost
+
+def test_call_method(agent, task_data):
+    """Test that the __call__ method invokes execute_task."""
+    mock_result = {
+        "configuration": {"eval_metric": "f1"},
+        "cost_summary": {"total_cost": 0.01}
+    }
+    with patch.object(AutoGluonConfigAgent, 'execute_task', return_value=mock_result) as mock_execute:
+        result = agent(task_data)
+        mock_execute.assert_called_once_with(task_data)
+        assert result == mock_result