llm4ad 1.0__tar.gz

llm4ad-1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm4ad-1.0/PKG-INFO

@@ -0,0 +1,18 @@
+Metadata-Version: 2.1
+Name: llm4ad
+Version: 1.0
+Summary: Large Language Model for Algorithm Design Platform 
+Author: Optima
+Requires-Python: >=3.9,<=3.11
+License-File: LICENSE
+Requires-Dist: numpy<2
+Requires-Dist: tensorboard
+Requires-Dist: scipy
+Requires-Dist: tqdm
+Requires-Dist: requests
+Requires-Dist: openai
+Requires-Dist: pytz
+Requires-Dist: matplotlib
+Requires-Dist: python-docx
+Requires-Dist: gym
+Requires-Dist: ttkbootstrap

llm4ad-1.0/README.md

@@ -0,0 +1,306 @@
+
+
+<div align="center">
+<h1 align="center">
+<img src="./assets/figs/logo.png" alt="LLM4AD Logo" style="width: 90%; height: auto;">
+</h1>
+<h1 align="center">
+LLM4AD: Large Language Model for Algorithm Design
+</h1>
+
+[![Releases](https://img.shields.io/badge/Release-v1.0-blue)](https://github.com/Optima-CityU/LLM4AD/releases)
+![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-brightgreen.svg)
+[![PR's Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/Optima-CityU/LLM4AD/pulls)
+![Python](https://img.shields.io/badge/Python-3.9+-blue)
+[![License](https://img.shields.io/badge/License-MIT-important)](https://github.com/Optima-CityU/LLM4AD/blob/main/LICENSE)
+[![Documentation Status](https://readthedocs.org/projects/llm4ad-doc/badge/?version=latest)](https://llm4ad-doc.readthedocs.io/en/latest/?badge=latest)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Optima-CityU/llm4ad/blob/main/example/online_bin_packing/online_bin_packing_tutorial.ipynb)
+
+[Website](http://www.llm4ad.com/)
+| [Documentation](https://llm4ad-doc.readthedocs.io/en/latest/)
+| [Examples](https://github.com/Optima-CityU/LLM4AD/tree/main/example)
+| [GUI](https://github.com/Optima-CityU/LLM4AD/tree/main/GUI)
+
+</div>
+<br>
+
+## Introduction 📖
+
+LLM4AD is an open-source Python-based Platform leveraging **Large Language Models (LLMs)** for **Automatic Algorithm Design (AD)**. Please refer to the [paper](https://arxiv.org/pdf/2412.17287) [LLM4AD] for detailed information, including the overview, methodology, and benchmark results.
+
+LLM4AD is accomplished with [Documents](https://llm4ad-doc.readthedocs.io/en/latest/) and [Examples](https://github.com/Optima-CityU/LLM4AD/tree/main/example) materials to support users and developers to easily test, build and deploy their own LLM4AD applications and conduct research.
+
+LLM4AD was originally developed for [optimisation tasks](https://llm4ad-doc.readthedocs.io/en/latest/task/optimization/index.html). The framework is versatile enough to be used in other areas, including [machine learning](https://llm4ad-doc.readthedocs.io/en/latest/task/machine_learning/index.html), [science discovery](https://llm4ad-doc.readthedocs.io/en/latest/task/science_discovery/index.html), game theory and engineering design.
+
+For more information, see the [contact list](https://github.com/Optima-CityU/LLM4AD#Contact)
+
+<img src="./assets/figs/overview.png" alt="llm4ad" style="zoom:50%;" />
+
+## 🔥 News 
+
++ 2024.12 🎉🎉 **LLM4AD paper Released** [“LLM4AD: A Platform for Algorithm Design with Large Language Model"](https://arxiv.org/pdf/2412.17287) ! 
+
++ 2024.11 🎉🎉 **LLM4AD v1.0 Released**  ! 
+
++ 2024.10 🎉🎉 **Survey Paper** [“A Systematic Survey on Large Language Models for Algorithm Design”](https://arxiv.org/pdf/2410.14716) is online ! 
+
+  
+
+## 💡 Features of our package
+
+| Feature                                                      | Support / To be supported |
+| ------------------------------------------------------------ | ------------------------- |
+| **Unified Interfaces** for methods                           | 🔥Support            |
+| **Unified Interfaces** for tasks                             | 🔥Support             |
+| **Unified Interfaces** for LLMs                              | 🔥Support             |
+| **Evaluation acceleration:** multiprocessing evaluation, add Numba wrapper for heuristic | 🔥Support              |
+| **Secure Evaluation:** main process protection, timeout interruption | 🔥Support             |
+| **Logs:** local logs, Wandb and Tensorboard support          | 🔥Support             |
+| **GUI:** methods selection, tasks selection, convergence, best algorithm, ... | 🔥Support             |
+| **Resume run**                                               | 🚀Will be updated soon     |
+| Support other languages                                      | 🚀Will be updated soon     |
+| More search methods                                          | 🚀Will be updated soon     |
+| More task examples                                           | 🚀Will be updated soon     |
+
+
+
+## 🎁 Requirements & Installation
+
+> [!Important]
+> The Python version **must** be larger or equal to Python 3.9.
+
+
+- refer to [requirements.txt](./requirements.txt)
+
+- Numba (if you want to use Numba accelerate)
+
+- Tensorboard (if you want to use a Tensorboard logger)
+
+- wandb (if you want to use wandb logger)
+
+- gym (if you want to try **Machine Learning** tasks)
+
+- pandas (if you want to try **Science Discovery** tasks)
+
+- all required packages in [requirements.txt](./requirements.txt) (if you want to use GUI)
+
+  #### Install LLM4AD locally
+
+  We suggest to install and run LLM4AD in [conda](https://conda.io/projects/conda/en/latest/index.html) env with python>=3.9, <=3.11
+
+  ```bash
+  cd LLM4AD
+  
+  pip install .
+  ```
+
+  #### Install LLM4AD using PiPy
+
+  We suggest to install and run LLM4AD in [conda](https://conda.io/projects/conda/en/latest/index.html) env with python>=3.9, <=3.11
+
+  ```bash
+  pip install llm4ad
+  ```
+
+  
+
+## 💻 Example Usage 
+
+### Quick Start:
+
+> [!Note]
+> Configure your LLM api before running the script. For example:
+>
+> 1) Set `host`: 'api.deepseek.com'
+> 2) Set `key`: 'your api key'
+> 3) Set `model` `deepseek-chat'
+
+```python
+from llm4ad.task.optimization.online_bin_packing import OBPEvaluation
+from llm4ad.tools.llm.llm_api_https import HttpsApi
+from llm4ad.method.eoh import EoH, EoHProfiler
+
+
+def main():
+
+    llm = HttpsApi(host="xxx",  # your host endpoint, e.g., api.openai.com, api.deepseek.com
+                   key="sk-xxx",  # your key, e.g., sk-xxxxxxxxxx
+                   model="xxx",  # your llm, e.g., gpt-3.5-turbo, deepseek-chat
+                   timeout=20)
+    
+    task = OBPEvaluation()
+
+    method = EoH(llm=llm,
+                 profiler=EoHProfiler(log_dir='logs/eoh', log_style='simple'),
+                 evaluation=task,
+                 max_sample_nums=20,
+                 max_generations=10,
+                 pop_size=4,
+                 num_samplers=1,
+                 num_evaluators=1, 
+                 debug_mode=False)
+
+    method.run()
+
+
+if __name__ == '__main__':
+    main()
+
+```
+
+
+### More Examples:
++ [Constructive Heuristics for TSP](https://github.com/Optima-CityU/LLM4AD/blob/main/example/tsp_construct/run_eoh.py)
++ [Constructive Heuristics for VRPTW](https://github.com/Optima-CityU/LLM4AD/blob/main/example/vrptw_construct/run_eoh.py)
++ ...
+  
+
+Check [Documents](https://llm4ad-doc.readthedocs.io/en/latest/index.html) for more tasks and examples
+
+### GUI usage:
+
+> [!Important]
+> Install all required packages in [requirements.txt](./requirements.txt) for GUI usage
+
+```bash
+$ cd GUI
+$ python run_gui.py
+```
+
+Check [GUI Introduction](https://llm4ad-doc.readthedocs.io/en/latest/getting_started/gui.html) for more information
+
+<img src="./assets/figs/gui.gif" alt="llm4ad" style="zoom:80%;" />
+
+## 📦 LLM4AD Search Methods
+
+| Methods                                               | Paper title                                                  |
+| ----------------------------------------------------- | ------------------------------------------------------------ |
+| **AEL** (Preliminary Version of **EoH**)        | [Algorithm Evolution using Large Language Model](https://arxiv.org/abs/2311.15249) (Arxiv 2023) |
+| **EoH**        | [Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model](https://openreview.net/pdf?id=BwAkaxqiLB) (ICML 2024) |
+| **FunSearch**                                             | [Mathematical Discoveries from Program Search with Large Language Models](https://www.nature.com/articles/s41586-023-06924-6) (Nature 2024) |
+| **RandomSampling**                                        | [Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Models](https://arxiv.org/abs/2407.10873) (PPSN 2024) |
+| **(1+1)-EPS** <br/>(HillClimbing) | [Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Models](https://arxiv.org/abs/2407.10873) (PPSN 2024) |
+| RegEvo                                                | coming soon                                                 |
+| Neighborhood search methods                           | coming soon                                                 |
+| Multi-objective search methods                        | coming soon                                                 |
+| Others                                                | coming soon                                                 |
+
+
+## 📦LLM4AD Algorithm Design Tasks 
+
+
+| Area              | Algorithm Task                                               | Paper                                             |
+| ----------------- | ------------------------------------------------------------ | ------------------------------------------------- |
+| Optimization      | [Online Bin Packing, Constructive heuristic]                 | [paper](https://openreview.net/pdf?id=BwAkaxqiLB) |
+|                   | [Traveling Salesman Problem (TSP), Construct heuristic](https://llm4ad-doc.readthedocs.io/en/latest/task/optimization/tsp_construct.html) | [paper](https://arxiv.org/pdf/2311.15249)         |
+|                   | Traveling Salesman Problem (TSP), Guided local search        | [paper](https://openreview.net/pdf?id=BwAkaxqiLB) |
+|                   | Flow Shop Scheduling Problem (FSSP), Guided local search     | [paper](https://openreview.net/pdf?id=BwAkaxqiLB) |
+|                   | Bayesian Optimization, Cost-aware Acquisition Function Design | [paper](https://arxiv.org/abs/2404.16906)         |
+| Machine Learning  | Adversarial Attack, Attack strategy                          | [paper](https://arxiv.org/abs/2401.15335)         |
+|                   | [Acrobot, Heuristic (Agent)](https://llm4ad-doc.readthedocs.io/en/latest/task/machine_learning/acrobot.html) |                                                   |
+|                   | [Cart Pole, Heuristic (Agent)](https://llm4ad-doc.readthedocs.io/en/latest/task/machine_learning/car_pole.html) |                                                   |
+|                   | [Mountain Car, Heuristic (Agent)](https://llm4ad-doc.readthedocs.io/en/latest/task/machine_learning/mountain_car.html) |                                                   |
+| Science Discovery | Computational fluid dynamics, Turbulence model design        | [paper](https://arxiv.org/pdf/2410.10657)         |
+|                   | [Bacteria Growth, Function](https://llm4ad-doc.readthedocs.io/en/latest/task/science_discovery/bacteria_grow.html) |                                                   |
+|                   | [Oscillator, Equation](https://llm4ad-doc.readthedocs.io/en/latest/task/science_discovery/oscillator1.html) |                                                   |
+|                   | [Stress & Strain, Equation](https://llm4ad-doc.readthedocs.io/en/latest/task/science_discovery/stress_strain.html) |                                                   |
+| Math              | Admissible Sets                                              | [paper](https://www.nature.com/articles/s41586-023-06924-6)                                                  |
+| coming soon ...   |                                                              |                                                   |
+
+
+
+## 🤖 LLM Interfaces
+There are three approaches on LLM interface implementation, check [Tutorial on LLM interface implementation](https://llm4ad-doc.readthedocs.io/en/latest/dev/llm.html) for more information.
++ **Remote LLM API** (e.g., GPT4o, GPT3.5, Gemini Pro, Deepseek ...) (**<Recommended !!!>**)
++ **Local HuggingFace LLM Deployment** (e.g., Llamacode, Llama, Gemma, Deepseek, ...)
++ **Your Implementation** If you want to use your own GPT API or local LLMs deployment, please create and add your interface in [LLM](https://github.com/Optima-CityU/LLM4AD/tree/main/llm4ad/tools/llm)
+
+
+
+## 🗎 Tutorial: How to Use LLM4AD to Solve Your Algorithm Design Task
+
+A Step-by-step Tutorial on using LLM4AD to solve your algorithm design task is provided [here](https://llm4ad-doc.readthedocs.io/en/latest/dev/run_new_task.html#)
+
+
+
+
+## 🪪 Licence
+
+This project is licensed under the **MIT License** - see the [LICENSE](./LICENSE) file for details. Parts of this project use code licensed under the Apache License 2.0.
+
+
+
+## ✨Reference 
+
+If you find LLM4AD helpful please cite:
+
+```bibtex
+@article{liu2024llm4ad,
+      title={LLM4AD: A Platform for Algorithm Design with Large Language Model}, 
+      author={Fei Liu and Rui Zhang and Zhuoliang Xie and Rui Sun and Kai Li and Xi Lin and Zhenkun Wang and Zhichao Lu and Qingfu Zhang},
+      year={2024},
+      eprint={2412.17287},
+      archivePrefix={arXiv},
+      primaryClass={cs.AI},
+      url={https://arxiv.org/abs/2412.17287}, 
+}
+```
+
+
+
+## About LLM4AD
+
+This platform is developed and maintained by LLM4AD developer group from the City University of Hong Kong (CityUHK) and the Southern University of Science and Technology (SUSTech). We develop LLM4AD platform for research purposes and hope to contribute to the research area by delivering tools for LLM-based algorithm design methods.
+
++ **Contribution:** We are more than welcome to contribute (see our 📖 [contribution guide](https://github.com/Optima-CityU/llm4ad/tree/main/assets/contribution.md)) including developing code and ideas to improve our platform.
++ **Collaborations:** If you like our platform, and you would like to use it for profit-making purposes? We are always searching for industrial collaborations because they help direct research to meet the industry’s needs.
++ **Issue:** If you find a bug or you have any kind of concern regarding the correctness, please report us an [issue](https://github.com/Optima-CityU/llm4ad/issues).
++ **Profit Purpose:** If you intend to use LLM4AD for any profit-making purposes, please contact [us](http://www.llm4ad.com/contact.html).
+
+## :star: Star Trends
+[![Star History Chart](https://api.star-history.com/svg?repos=Optima-CityU/llm4ad&type=Date)](https://star-history.com/#Optima-CityU/llm4ad&Date)
+
+## Contact
+
+If you are interested in LLM4AD or if you encounter any difficulty using the platform, you can:
+
+1. Visit our website [LLM4AD Web](http://www.llm4ad.com)
+
+2. Visit our collection [a collection of resources and research papers on LLM4AD](https://github.com/FeiLiu36/LLM4Opt)
+
+3. Visit [Discussions](https://github.com/Optima-CityU/llm4ad/discussions) to connect with other members of our community
+
+4. Join our QQ Group
+
+   <img src="./assets/figs/qq.png" alt="LLM4AD Logo" style="width: 30%; height: auto;">
+
+5. Contact us through email fliu36-c@my.cityu.edu.hk
+
+6. Submit an [issue](https://github.com/Optima-CityU/LLM4AD)
+
+
+
+## Contributors
+
+Any new ideas, features, and improvements are welcomed!
+
+You can contribute to LLM4AD follow our 📖 [Contribution Guide](https://github.com/Optima-CityU/llm4ad/tree/main/assets/contribution.md).
+
+Thank you for contributing to LLM4AD and welcome to being part of the LLM4AD community! :sparkles:
+
+<table>
+  <tbody>
+    <tr>
+      <td align="center" valign="top" width="12.5%"><a href="https://github.com/FeiLiu36"><img src="https://avatars.githubusercontent.com/FeiLiu36" width="50px;" alt="Fei Liu"/><br /><sub><b>Fei Liu</b></sub></td>
+<td align="center" valign="top" width="12.5%"><a href="https://ahalikai.github.io/"><img src="https://github.com/Ahalikai/Ahalikai.github.io/blob/main/Aha.jpg?raw=true" width="50px;" alt="Kai Li"/><br /><sub><b>Kai Li</b></sub>
+        </td>
+      <td align="center" valign="top" width="12.5%"><a href="https://github.com/SunnyR7"><img src="https://avatars.githubusercontent.com/SunnyR7" width="50px;" alt="Rui Sun"/><br /><sub><b>Rui Sun</b></sub></td>
+      <td align="center" valign="top" width="12.5%"><a href="https://github.com/Acquent0"><img src="https://avatars.githubusercontent.com/Acquent0" width="50px;" alt="Julian XIE"/><br  /><sub><b>Julian XIE</b></sub>
+        </td>
+ <td align="center" valign="top" width="12.5%"><a href="https://github.com/ShunyuYao6"><img src="https://avatars.githubusercontent.com/ShunyuYao6" width="50px;" alt="Shunyu Yao"/><br /><sub><b>Shunyu Yao</b></sub>
+        </td>
+     <td align="center" valign="top" width="12.5%"><a href="https://github.com/RayZhhh"><img src="https://avatars.githubusercontent.com/RayZhhh" width="50px;" alt="Rui Zhang"/><br /><sub><b>Rui Zhang</b></sub>
+        </td>
+    </tr>
+
+

llm4ad-1.0/llm4ad/__init__.py

@@ -0,0 +1,6 @@
+from . import base
+from . import method
+from . import task
+from .tools import profiler
+from .tools import llm
+

llm4ad-1.0/llm4ad/base/__init__.py

@@ -0,0 +1,9 @@
+from . import code, evaluate, sample, modify_code
+from .code import (
+    Function,
+    Program,
+    TextFunctionProgramConverter
+)
+from .evaluate import Evaluation, SecureEvaluator
+from .modify_code import ModifyCode
+from .sample import LLM, SampleTrimmer

llm4ad-1.0/llm4ad/base/code.py

@@ -0,0 +1,270 @@
+"""
+This file implements 2 classes representing unities of code:
+
+- Function, containing all the information we need about functions: name, args,
+  body and optionally a return type and a docstring.
+
+- Program, which contains a code preface (which could be imports, global
+  variables and classes, ...) and a list of Functions.
+
+- For example, a function is shown below,
+which is an un-executable program because 'np' is not defined, and 'WEIGHT' is not defined.
+--------------------------------------------------------------------------------------------
+def func(a: np.ndarray, b: np.ndarray) -> np.ndarray:
+    b = b + WEIGHT
+    return a + b
+--------------------------------------------------------------------------------------------
+
+- A program is an executable object as shown below.
+--------------------------------------------------------------------------------------------
+import numpy as np
+WEIGHT = 10
+
+def func(a: np.ndarray, b: np.ndarray) -> np.ndarray:
+    b = b + WEIGHT
+    return a + b
+--------------------------------------------------------------------------------------------
+"""
+from __future__ import annotations
+
+import ast
+import copy
+import dataclasses
+from typing import Any
+
+
+@dataclasses.dataclass
+class Function:
+    """A parsed Python function."""
+
+    name: str
+    args: str
+    body: str
+    return_type: str | None = None
+    docstring: str | None = None
+    score: Any | None = None
+    evaluate_time: float | None = None
+    sample_time: float | None = None
+
+    def __str__(self) -> str:
+        return_type = f' -> {self.return_type}' if self.return_type else ''
+
+        function = f'def {self.name}({self.args}){return_type}:\n'
+        if self.docstring:
+            # self.docstring is already indented on every line except the first one.
+            # Here, we assume the indentation is always four spaces.
+            new_line = '\n' if self.body else ''
+            function += f'    """{self.docstring}"""{new_line}'
+        # self.body is already indented.
+        function += self.body + '\n\n'
+        return function
+
+    def __setattr__(self, name: str, value: str) -> None:
+        # Ensure there aren't leading & trailing new lines in `body`.
+        if name == 'body':
+            value = value.strip('\n')
+        # Ensure there aren't leading & trailing quotes in `docstring``.
+        if name == 'docstring' and value is not None:
+            if '"""' in value:
+                value = value.strip()
+                value = value.replace('"""', '')
+        super().__setattr__(name, value)
+
+    def __eq__(self, other: Function):
+        assert isinstance(other, Function)
+        return (self.name == other.name and
+                self.args == other.args and
+                self.return_type == other.return_type and
+                self.body == other.body)
+
+
+@dataclasses.dataclass(frozen=True)
+class Program:
+    """A parsed Python program."""
+
+    # `preface` is everything from the beginning of the code till the first
+    # function is found.
+    preface: str
+    functions: list[Function]
+
+    def __str__(self) -> str:
+        program = f'{self.preface}\n' if self.preface else ''
+        program += '\n'.join([str(f) for f in self.functions])
+        return program
+
+    def find_function_index(self, function_name: str) -> int:
+        """Returns the index of input function name."""
+        function_names = [f.name for f in self.functions]
+        count = function_names.count(function_name)
+        if count == 0:
+            raise ValueError(
+                f'function {function_name} does not exist in program:\n{str(self)}'
+            )
+        if count > 1:
+            raise ValueError(
+                f'function {function_name} exists more than once in program:\n'
+                f'{str(self)}'
+            )
+        index = function_names.index(function_name)
+        return index
+
+    def get_function(self, function_name: str) -> Function:
+        index = self.find_function_index(function_name)
+        return self.functions[index]
+
+
+class _ProgramVisitor(ast.NodeVisitor):
+    """Parses code to collect all required information to produce a `Program`.
+    Note that we do not store function decorators.
+    """
+
+    def __init__(self, sourcecode: str):
+        self._codelines: list[str] = sourcecode.splitlines()
+        self._preface: str = ''
+        self._functions: list[Function] = []
+        self._current_function: str | None = None
+
+    def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
+        """Collects all information about the function being parsed.
+        """
+        # We only care about first level functions.
+        if node.col_offset == 0:
+            self._current_function = node.name
+            # --------------------------------------------------------------------------------------------
+            # RZ: Here is an issue: some cases were not taken into consideration.
+            # For instance, if the first parsed function has decorators,
+            # these decorators will also be retained in the preceding code snippet.
+            # However, in the algorithm's flow, these decorators are invalid,
+            # leading to subsequent evaluations reporting errors.
+            # --------------------------------------------------------------------------------------------
+            # if not self._functions:
+            #     self._preface = '\n'.join(self._codelines[:node.lineno - 1])
+            # --------------------------------------------------------------------------------------------
+            # Fix bugs: if the first function is decorated, find the smallest line_no of the decorator,
+            # FIX bugs: and retrain the code above it as 'preface'.
+            # --------------------------------------------------------------------------------------------
+            if not self._functions:
+                has_decorators = bool(node.decorator_list)
+                if has_decorators:
+                    # find the minimum line number and retain the code above
+                    decorator_start_line = min(decorator.lineno for decorator in node.decorator_list)
+                    self._preface = '\n'.join(self._codelines[:decorator_start_line - 1])
+                else:
+                    self._preface = '\n'.join(self._codelines[:node.lineno - 1])
+            # --------------------------------------------------------------------------------------------
+            function_end_line = node.end_lineno
+            body_start_line = node.body[0].lineno - 1
+            # Extract the docstring.
+            docstring = None
+            if isinstance(node.body[0], ast.Expr) and isinstance(node.body[0].value, ast.Str):
+                docstring = f'    """{ast.literal_eval(ast.unparse(node.body[0]))}"""'
+                if len(node.body) > 1:
+                    # ----------------------------------------------------------------------------
+                    # RZ: If the next line of the docstring is also comment that starts with "#"
+                    # We should preserve it.
+                    # ----------------------------------------------------------------------------
+                    # body_start_line = node.body[1].lineno - 1
+                    # ----------------------------------------------------------------------------
+                    # Fix bugs: If we find the docstring, the function body start with
+                    # Fix bugs: body[0].end_lineno, where body[0] refers to the docstring node.
+                    # ----------------------------------------------------------------------------
+                    body_start_line = node.body[0].end_lineno
+                    # ----------------------------------------------------------------------------
+                else:
+                    body_start_line = function_end_line
+
+            self._functions.append(Function(
+                name=node.name,
+                args=ast.unparse(node.args),
+                return_type=ast.unparse(node.returns) if node.returns else None,
+                docstring=docstring,
+                body='\n'.join(self._codelines[body_start_line:function_end_line]),
+            ))
+        self.generic_visit(node)
+
+    def return_program(self) -> Program:
+        return Program(preface=self._preface, functions=self._functions)
+
+
+class TextFunctionProgramConverter:
+    """Convert text to Program instance and Function instance,
+    Convert Program instance to Function instance, and Function instance to Program instance.
+    """
+
+    @classmethod
+    def text_to_program(cls, program_str: str) -> Program | None:
+        """Returns Program object by parsing input text using Python AST.
+        """
+        try:
+            # We assume that the program is composed of some preface (e.g. imports,
+            # classes, assignments, ...) followed by a sequence of functions.
+            tree = ast.parse(program_str)
+            visitor = _ProgramVisitor(program_str)
+            visitor.visit(tree)
+            return visitor.return_program()
+        except:
+            return None
+
+    @classmethod
+    def text_to_function(cls, program_str: str) -> Function | None:
+        """Returns Function object by parsing input text using Python AST.
+        """
+        try:
+            program = cls.text_to_program(program_str)
+            if len(program.functions) != 1:
+                raise ValueError(f'Only one function expected, got {len(program.functions)}'
+                                 f':\n{program.functions}')
+            return program.functions[0]
+        except ValueError as value_err:
+            raise value_err
+        except:
+            return None
+
+    @classmethod
+    def function_to_program(cls, function: str | Function, template_program: str | Program) -> Program | None:
+        try:
+            # convert function to Function instance
+            if isinstance(function, str):
+                function = cls.text_to_function(function)
+            else:
+                function = copy.deepcopy(function)
+
+            # convert template_program to Program instance
+            if isinstance(template_program, str):
+                template_program = cls.text_to_program(template_program)
+            else:
+                template_program = copy.deepcopy(template_program)
+
+            # assert that a program have one function
+            if len(template_program.functions) != 1:
+                raise ValueError(f'Only one function expected, got {len(template_program.functions)}'
+                                 f':\n{template_program.functions}')
+
+            # replace the function body with the new function body
+            template_program.functions[0].body = function.body
+            return template_program
+        except ValueError as value_err:
+            raise value_err
+        except:
+            return None
+
+    @classmethod
+    def program_to_function(cls, program: str | Program) -> Function | None:
+        try:
+            # convert program to Program instance
+            if isinstance(program, str):
+                program = cls.text_to_program(program)
+            else:
+                program = copy.deepcopy(program)
+
+            # assert that a program have one function
+            if len(program.functions) != 1:
+                raise ValueError(f'Only one function expected, got {len(program.functions)}'
+                                 f':\n{program.functions}')
+
+            # return the function
+            return program.functions[0]
+        except ValueError as value_err:
+            raise value_err
+        except:
+            return None