autohd 0.1.0__tar.gz

autohd-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 AutoHD Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

autohd-0.1.0/PKG-INFO

@@ -0,0 +1,192 @@
+Metadata-Version: 2.4
+Name: autohd
+Version: 0.1.0
+Summary: Automated Heuristics Discovery for LLM Planning - Enable LLMs to generate and evolve heuristic functions for complex planning tasks
+Author-email: Your Name <your.email@example.com>
+License: MIT
+Project-URL: Homepage, https://github.com/yourusername/autohd
+Project-URL: Documentation, https://github.com/yourusername/autohd#readme
+Project-URL: Repository, https://github.com/yourusername/autohd
+Project-URL: Issues, https://github.com/yourusername/autohd/issues
+Project-URL: Bug Tracker, https://github.com/yourusername/autohd/issues
+Keywords: llm,planning,heuristics,reasoning,inference-time,autohd,test-time-compute
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Mathematics
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Requires-Dist: anthropic>=0.20.0
+Requires-Dist: litellm>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+Requires-Dist: black>=24.0.0; extra == "dev"
+Requires-Dist: ruff>=0.3.0; extra == "dev"
+Requires-Dist: mypy>=1.8.0; extra == "dev"
+Requires-Dist: httpx>=0.27.0; extra == "dev"
+Provides-Extra: cli
+Requires-Dist: click>=8.0.0; extra == "cli"
+Requires-Dist: rich>=13.0.0; extra == "cli"
+Provides-Extra: langchain
+Requires-Dist: langchain>=0.1.0; extra == "langchain"
+Requires-Dist: langchain-core>=0.1.0; extra == "langchain"
+Requires-Dist: langchain-openai>=0.0.5; extra == "langchain"
+Provides-Extra: all
+Requires-Dist: autohd[cli,dev,langchain]; extra == "all"
+Dynamic: license-file
+
+# AutoHD: Automated Heuristics Discovery for LLM Planning
+
+[![PyPI](https://img.shields.io/pypi/v/autohd)](https://pypi.org/project/autohd/)
+[![Python](https://img.shields.io/pypi/pyversions/autohd)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/pypi/l/autohd)](https://github.com/yourusername/autohd/blob/main/LICENSE)
+[![Documentation](https://img.shields.io/badge/docs-User%20Guide-blue)](https://github.com/yourusername/autohd#readme)
+
+---
+
+## What is AutoHD?
+
+**AutoHD** (Automated Heuristics Discovery) is a Python package that enables LLMs to generate and evolve heuristic functions for guiding inference-time search in complex planning tasks.
+
+Based on the research paper: ["Complex LLM Planning via Automated Heuristics Discovery"](https://arxiv.org/abs/2502.19295v1) (arXiv:2502.19295v1, Feb 2025, Texas A&M University).
+
+### Key Features
+
+- **LLM-Generated Heuristics**: LLMs automatically generate heuristic functions as Python code
+- **Heuristic Evolution**: Iterative refinement through exploration and modification strategies
+- **Multiple Search Algorithms**: Support for Greedy BFS and A* search
+- **Multiple Planning Tasks**: Blocksworld, Game of 24, and Rubik's Cube
+- **LLM-Agnostic**: Works with OpenAI, Anthropic, and other providers via litellm
+- **LangChain Integration**: Plug into LangChain agents and chains
+
+## Quick Start
+
+```bash
+pip install autohd
+```
+
+### Basic Usage
+
+```python
+import asyncio
+from autohd import AutoHD
+from autohd.core.config import AutoHDConfig, LLMConfig
+
+async def main():
+    # Configure AutoHD
+    config = AutoHDConfig(
+        task_name="blocksworld",
+        llm=LLMConfig(model="gpt-4o", provider="openai"),
+    )
+    
+    # Initialize AutoHD
+    autohd = AutoHD(config=config)
+    
+    # Generate synthetic training data
+    train_data = [
+        {"initial_state": "the red block is clear...", "goal_state": "the orange block is on top of the red block..."},
+        # ... more examples
+    ]
+    
+    # Run the pipeline
+    results = await autohd.run(
+        train_data=train_data,
+        test_data=test_data,
+    )
+    
+    print(f"Best heuristic accuracy: {results['test_results']['accuracy']:.1%}")
+
+asyncio.run(main())
+```
+
+### CLI Usage
+
+```bash
+# Run on Blocksworld
+autohd run --task blocksworld --model gpt-4o
+
+# Run on Game of 24
+autohd run --task game24 --model gpt-4o
+
+# Run on Rubik's Cube
+autohd run --task rubiks_cube --model gpt-4o
+
+# Custom configuration
+autohd run --task blocksworld --generations 5 --population 10 --search-algorithm astar
+```
+
+## Architecture
+
+```
+autohd/
+├── core/           # Core framework
+│   ├── autohd.py   # Main AutoHD class
+│   ├── config.py   # Configuration classes
+│   └── llm_client.py # LLM API abstraction
+├── search/         # Search algorithms
+│   ├── greedy_bfs.py # Greedy BFS solver
+│   └── astar.py     # A* solver
+├── benchmarks/     # Planning task implementations
+│   ├── base.py     # Base benchmark class
+│   ├── blocksworld.py # Blocksworld task
+│   ├── game24.py   # Game of 24 task
+│   └── rubiks_cube.py # Rubik's Cube task
+├── integrations/   # Framework integrations
+│   └── langchain.py # LangChain integration
+└── cli/            # Command-line interface
+    └── main.py     # CLI entry point
+```
+
+## How It Works
+
+AutoHD follows a 5-step pipeline:
+
+1. **Generate Initial Heuristics**: LLMs are prompted to create heuristic functions as Python code
+2. **Evaluate Heuristics**: Each heuristic is tested on validation data using heuristic-guided search
+3. **Evolve Heuristics**: Iterative refinement through exploration (new ideas) and modification (tweaking existing)
+4. **Select Best**: The best heuristic from all generations is chosen
+5. **Test**: The selected heuristic is evaluated on test data
+
+## Performance
+
+Based on the research paper:
+
+| Task | GPT-4o-mini | GPT-4o | Llama 3.1 70B | Best Baseline |
+|------|-------------|--------|---------------|---------------|
+| Blocksworld | 42.4% | **75.1%** | 59.1% | ~2× improvement |
+| Game of 24 | 54% | **70%** | 69% | vs ToT 42/62/59 |
+| Rubik's Cube (2×2) | **82.5%** | **83.1%** | **84.7%** | vs XoT 67/80/78 |
+
+## Requirements
+
+- Python 3.10+
+- OpenAI API key (or other LLM provider)
+- Optional: `litellm` for multi-provider support
+
+## License
+
+MIT License
+
+## Citation
+
+If you use AutoHD in your research, please cite:
+
+```
+@article{ling2025autohd,
+  title={Complex LLM Planning via Automated Heuristics Discovery},
+  author={Ling, Hongyi and Parashar, Shubham and Khurana, Sambhav and Olson, Blake and Basu, Anwesha and Sinha, Gaurangi and Tu, Zhengzhong and Caverlee, James and Ji, Shuiwang},
+  journal={arXiv preprint arXiv:2502.19295},
+  year={2025}
+}
+```

autohd-0.1.0/README.md

@@ -0,0 +1,144 @@
+# AutoHD: Automated Heuristics Discovery for LLM Planning
+
+[![PyPI](https://img.shields.io/pypi/v/autohd)](https://pypi.org/project/autohd/)
+[![Python](https://img.shields.io/pypi/pyversions/autohd)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/pypi/l/autohd)](https://github.com/yourusername/autohd/blob/main/LICENSE)
+[![Documentation](https://img.shields.io/badge/docs-User%20Guide-blue)](https://github.com/yourusername/autohd#readme)
+
+---
+
+## What is AutoHD?
+
+**AutoHD** (Automated Heuristics Discovery) is a Python package that enables LLMs to generate and evolve heuristic functions for guiding inference-time search in complex planning tasks.
+
+Based on the research paper: ["Complex LLM Planning via Automated Heuristics Discovery"](https://arxiv.org/abs/2502.19295v1) (arXiv:2502.19295v1, Feb 2025, Texas A&M University).
+
+### Key Features
+
+- **LLM-Generated Heuristics**: LLMs automatically generate heuristic functions as Python code
+- **Heuristic Evolution**: Iterative refinement through exploration and modification strategies
+- **Multiple Search Algorithms**: Support for Greedy BFS and A* search
+- **Multiple Planning Tasks**: Blocksworld, Game of 24, and Rubik's Cube
+- **LLM-Agnostic**: Works with OpenAI, Anthropic, and other providers via litellm
+- **LangChain Integration**: Plug into LangChain agents and chains
+
+## Quick Start
+
+```bash
+pip install autohd
+```
+
+### Basic Usage
+
+```python
+import asyncio
+from autohd import AutoHD
+from autohd.core.config import AutoHDConfig, LLMConfig
+
+async def main():
+    # Configure AutoHD
+    config = AutoHDConfig(
+        task_name="blocksworld",
+        llm=LLMConfig(model="gpt-4o", provider="openai"),
+    )
+    
+    # Initialize AutoHD
+    autohd = AutoHD(config=config)
+    
+    # Generate synthetic training data
+    train_data = [
+        {"initial_state": "the red block is clear...", "goal_state": "the orange block is on top of the red block..."},
+        # ... more examples
+    ]
+    
+    # Run the pipeline
+    results = await autohd.run(
+        train_data=train_data,
+        test_data=test_data,
+    )
+    
+    print(f"Best heuristic accuracy: {results['test_results']['accuracy']:.1%}")
+
+asyncio.run(main())
+```
+
+### CLI Usage
+
+```bash
+# Run on Blocksworld
+autohd run --task blocksworld --model gpt-4o
+
+# Run on Game of 24
+autohd run --task game24 --model gpt-4o
+
+# Run on Rubik's Cube
+autohd run --task rubiks_cube --model gpt-4o
+
+# Custom configuration
+autohd run --task blocksworld --generations 5 --population 10 --search-algorithm astar
+```
+
+## Architecture
+
+```
+autohd/
+├── core/           # Core framework
+│   ├── autohd.py   # Main AutoHD class
+│   ├── config.py   # Configuration classes
+│   └── llm_client.py # LLM API abstraction
+├── search/         # Search algorithms
+│   ├── greedy_bfs.py # Greedy BFS solver
+│   └── astar.py     # A* solver
+├── benchmarks/     # Planning task implementations
+│   ├── base.py     # Base benchmark class
+│   ├── blocksworld.py # Blocksworld task
+│   ├── game24.py   # Game of 24 task
+│   └── rubiks_cube.py # Rubik's Cube task
+├── integrations/   # Framework integrations
+│   └── langchain.py # LangChain integration
+└── cli/            # Command-line interface
+    └── main.py     # CLI entry point
+```
+
+## How It Works
+
+AutoHD follows a 5-step pipeline:
+
+1. **Generate Initial Heuristics**: LLMs are prompted to create heuristic functions as Python code
+2. **Evaluate Heuristics**: Each heuristic is tested on validation data using heuristic-guided search
+3. **Evolve Heuristics**: Iterative refinement through exploration (new ideas) and modification (tweaking existing)
+4. **Select Best**: The best heuristic from all generations is chosen
+5. **Test**: The selected heuristic is evaluated on test data
+
+## Performance
+
+Based on the research paper:
+
+| Task | GPT-4o-mini | GPT-4o | Llama 3.1 70B | Best Baseline |
+|------|-------------|--------|---------------|---------------|
+| Blocksworld | 42.4% | **75.1%** | 59.1% | ~2× improvement |
+| Game of 24 | 54% | **70%** | 69% | vs ToT 42/62/59 |
+| Rubik's Cube (2×2) | **82.5%** | **83.1%** | **84.7%** | vs XoT 67/80/78 |
+
+## Requirements
+
+- Python 3.10+
+- OpenAI API key (or other LLM provider)
+- Optional: `litellm` for multi-provider support
+
+## License
+
+MIT License
+
+## Citation
+
+If you use AutoHD in your research, please cite:
+
+```
+@article{ling2025autohd,
+  title={Complex LLM Planning via Automated Heuristics Discovery},
+  author={Ling, Hongyi and Parashar, Shubham and Khurana, Sambhav and Olson, Blake and Basu, Anwesha and Sinha, Gaurangi and Tu, Zhengzhong and Caverlee, James and Ji, Shuiwang},
+  journal={arXiv preprint arXiv:2502.19295},
+  year={2025}
+}
+```

autohd-0.1.0/autohd/__init__.py

@@ -0,0 +1,24 @@
+"""AutoHD: Automated Heuristics Discovery for LLM Planning.
+
+A Python package that enables LLMs to generate and evolve heuristic functions
+for guiding inference-time search in complex planning tasks.
+
+Based on the research paper: "Complex LLM Planning via Automated Heuristics Discovery"
+(arXiv:2502.19295v1, Feb 2025, Texas A&M University).
+"""
+
+__version__ = "0.1.0"
+__author__ = "AutoHD Contributors"
+__email__ = "contact@autohd.ai"
+
+from autohd.core.autohd import AutoHD
+from autohd.core.config import AutoHDConfig, LLMConfig, SearchConfig, EvolutionConfig
+
+__all__ = [
+    "AutoHD",
+    "AutoHDConfig",
+    "LLMConfig",
+    "SearchConfig",
+    "EvolutionConfig",
+    "__version__",
+]

autohd-0.1.0/autohd/benchmarks/__init__.py

@@ -0,0 +1,41 @@
+"""Benchmark task implementations for AutoHD."""
+
+from autohd.benchmarks.base import BenchmarkTask
+from autohd.benchmarks.blocksworld import BlocksworldTask
+from autohd.benchmarks.game24 import Game24Task
+from autohd.benchmarks.rubiks_cube import RubiksCubeTask
+
+__all__ = [
+    "BenchmarkTask",
+    "BlocksworldTask",
+    "Game24Task",
+    "RubiksCubeTask",
+]
+
+
+def get_benchmark_task(task_name: str, **kwargs) -> BenchmarkTask:
+    """Factory function to get a benchmark task by name.
+    
+    Args:
+        task_name: Name of the task ('blocksworld', 'game24', 'rubiks_cube')
+        **kwargs: Arguments passed to the task constructor
+        
+    Returns:
+        An instance of the appropriate BenchmarkTask subclass
+        
+    Raises:
+        ValueError: If task_name is not recognized
+    """
+    task_map = {
+        "blocksworld": BlocksworldTask,
+        "game24": Game24Task,
+        "rubiks_cube": RubiksCubeTask,
+    }
+    
+    if task_name not in task_map:
+        raise ValueError(
+            f"Unknown task '{task_name}'. "
+            f"Available tasks: {list(task_map.keys())}"
+        )
+    
+    return task_map[task_name](task_name=task_name, **kwargs)

autohd-0.1.0/autohd/benchmarks/base.py

@@ -0,0 +1,134 @@
+"""Base class for benchmark tasks."""
+
+import time
+import logging
+from abc import ABC, abstractmethod
+from typing import List, Dict, Any, Callable, Optional
+
+from autohd.core.config import SearchConfig
+
+logger = logging.getLogger(__name__)
+
+
+class BenchmarkTask(ABC):
+    """Abstract base class for benchmark planning tasks.
+    
+    All planning tasks (Blocksworld, Game of 24, Rubik's Cube) should inherit from
+    this class and implement the required methods.
+    """
+    
+    def __init__(
+        self,
+        task_name: str,
+        heuristic_func: Optional[Callable] = None,
+        validation_set: Optional[List[Any]] = None,
+    ):
+        self.task_name = task_name
+        self.heuristic_func = heuristic_func
+        self.validation_set = validation_set or []
+        self.model = None  # LLM model for action generation (set externally)
+    
+    @abstractmethod
+    def state_representation(self) -> Any:
+        """Return the state representation for this task."""
+        pass
+    
+    @abstractmethod
+    def action_space(self, state: Any) -> List[str]:
+        """Return the valid actions for a given state."""
+        pass
+    
+    @abstractmethod
+    def transition(self, state: Any, action: str) -> Any:
+        """Execute an action and transition to the next state."""
+        pass
+    
+    @abstractmethod
+    def is_goal(self, state: Any) -> bool:
+        """Check if a state is a goal state."""
+        pass
+    
+    @abstractmethod
+    def heuristic(self, state: Any) -> float:
+        """Evaluate the heuristic value of a state."""
+        pass
+    
+    async def run(
+        self,
+        test_data: List[Any],
+        search_config: Optional[SearchConfig] = None,
+        model=None,
+    ) -> Dict[str, Any]:
+        """Run the benchmark on test data.
+        
+        Args:
+            test_data: List of test cases/problem instances
+            search_config: Configuration for search algorithm
+            model: LLM model for action generation (if not set during initialization)
+            
+        Returns:
+            Dictionary with accuracy, timing, and other metrics
+        """
+        if model:
+            self.model = model
+        
+        if search_config is None:
+            from autohd.core.config import SearchConfig
+            search_config = SearchConfig()
+        
+        start_time = time.time()
+        successful = 0
+        total = len(test_data)
+        errors = []
+        
+        for i, problem in enumerate(test_data):
+            try:
+                result = await self._solve_problem(problem, search_config)
+                if result.get("success", False):
+                    successful += 1
+            except Exception as e:
+                errors.append({"problem_index": i, "error": str(e)})
+                logger.warning(f"Error solving problem {i}: {e}")
+        
+        elapsed = time.time() - start_time
+        
+        return {
+            "accuracy": successful / total if total > 0 else 0,
+            "successful": successful,
+            "total": total,
+            "errors": errors,
+            "error_rate": len(errors) / total if total > 0 else 0,
+            "time_seconds": elapsed,
+            "avg_time_per_problem": elapsed / total if total > 0 else 0,
+        }
+    
+    async def _solve_problem(
+        self,
+        problem: Any,
+        search_config: SearchConfig,
+    ) -> Dict[str, Any]:
+        """Solve a single problem using heuristic-guided search.
+        
+        Uses either Greedy BFS or A* based on configuration.
+        """
+        from autohd.search import greedy_bfs, astar_search
+        
+        if search_config.algorithm == "greedy_bfs":
+            result = await greedy_bfs.solve(
+                problem=problem,
+                heuristic_func=self.heuristic_func,
+                max_steps=search_config.max_steps,
+                timeout=search_config.timeout_seconds,
+            )
+        else:  # astar
+            result = await astar_search.solve(
+                problem=problem,
+                heuristic_func=self.heuristic_func,
+                max_steps=search_config.max_steps,
+                timeout=search_config.timeout_seconds,
+            )
+        
+        return result
+    
+    def __repr__(self):
+        return f"{self.__class__.__name__}(task={self.task_name})"