useful-optimizer 0.1.1__tar.gz

useful_optimizer-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Anselm Hahn
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

useful_optimizer-0.1.1/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.1
+Name: useful-optimizer
+Version: 0.1.1
+Summary: A dedicated set of optimization algorithms for numeric problems.
+License: MIT
+Author: Anselm Hahn
+Author-email: Anselm.Hahn@gmail.com
+Requires-Python: >=3.10,<3.13
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: numpy (>=1.26.4,<2.0.0)
+Requires-Dist: scikit-learn (>=1.5.1,<2.0.0)
+Requires-Dist: scipy (>=1.12.0,<2.0.0)
+Description-Content-Type: text/markdown
+
+# Useful Optimizer
+
+Useful Optimizer is a dedicated set of optimization algorithms for numeric problems. It's designed to provide a comprehensive collection of optimization techniques that can be easily used and integrated into any project.
+
+## Version
+
+The current version of Useful Optimizer is 0.1.0.
+
+## Features
+
+- A wide range of optimization algorithms.
+- Easy to use and integrate.
+- Suitable for various numeric problems.
+- Having fun to play with the algorithms
+
+## Installation
+
+To install Useful Optimizer, you can use pip:
+
+```bash
+pip install git+https://github.com/Anselmoo/useful-optimizer
+```
+
+## Usage
+
+Here's a basic example of how to use Useful Optimizer:
+
+```python
+from opt.benchmark import CrossEntropyMethod
+from opt.benchmark.functions import shifted_ackley
+
+optimizer = CrossEntropyMethod(
+        func=shifted_ackley,
+        dim=2,
+        lower_bound=-12.768,
+        upper_bound=+12.768,
+        population_size=100,
+        max_iter=1000,
+    )
+best_solution, best_fitness = optimizer.search()
+print(f"Best solution: {best_solution}")
+print(f"Best fitness: {best_fitness}")
+```
+
+## Current Implemented Optimizer
+
+The current version of Useful Optimizer includes a wide range of optimization algorithms, each implemented as a separate module. Here's a brief overview of the implemented optimizers:
+
+Sure, here's a brief description of each optimizer:
+
+- **Adadelta, Adagrad, and Adaptive Moment Estimation**: These are gradient-based optimization algorithms commonly used in machine learning and deep learning.
+
+- **Ant Colony Optimization**: This is a nature-inspired algorithm that mimics the behavior of ants to solve optimization problems.
+
+- **Artificial Fish Swarm Algorithm**: This algorithm simulates the behavior of fish in nature to perform global optimization.
+
+- **Augmented Lagrangian Method**: This is a method to solve constrained optimization problems. It combines the objective function and the constraints into a single function using Lagrange multipliers.
+
+- **Bat Algorithm**: This is a metaheuristic optimization algorithm inspired by the echolocation behavior of microbats.
+
+- **Bee Algorithm**: This is a population-based search algorithm inspired by the food foraging behavior of honey bee colonies.
+
+- **Cat Swarm Optimization**: This algorithm is based on the behavior of cats and distinguishes between two forms of behavior in cats: seeking mode and tracing mode.
+
+- **CMA-ES (Covariance Matrix Adaptation Evolution Strategy)**: This is an evolutionary algorithm for difficult non-linear non-convex optimization problems in continuous domain.
+
+- **Colliding Bodies Optimization**: This is a physics-inspired optimization method, based on the collision and explosion of bodies.
+
+- **Cross Entropy Method**: This is a Monte Carlo method for importance sampling and optimization.
+
+- **Cuckoo Search**: This is a nature-inspired metaheuristic optimization algorithm, which is based on the obligate brood parasitism of some cuckoo species.
+
+- **Cultural Algorithm**: This is a type of evolutionary algorithm that is based on the concept of culture, or shared information and knowledge.
+
+- **Differential Evolution**: This is a population-based metaheuristic algorithm, which uses mechanisms inspired by biological evolution, such as reproduction, mutation, recombination, and selection.
+
+- **Eagle Strategy**: This is a metaheuristic optimization algorithm inspired by the hunting behavior of eagles.
+
+- **Estimation of Distribution Algorithm**: This is a stochastic optimization algorithm that uses a probabilistic model of candidate solutions.
+
+- **Firefly Algorithm**: This is a nature-inspired metaheuristic optimization algorithm, based on the flashing behavior of fireflies.
+
+- **Genetic Algorithm**: This is a search heuristic that is inspired by Charles Darwin’s theory of natural evolution.
+
+- **Glowworm Swarm Optimization**: This is a nature-inspired optimization algorithm based on the behavior of glowworms.
+
+- **Grey Wolf Optimizer**: This is a metaheuristic optimization algorithm inspired by grey wolves.
+
+- **Harmony Search**: This is a music-inspired metaheuristic optimization algorithm.
+
+- **Hessian Free Optimization**: This is a second-order optimization algorithm that uses information from the Hessian matrix to guide the search.
+
+- **Hill Climbing**: This is a mathematical optimization technique which belongs to the family of local search.
+
+- **Imperialist Competitive Algorithm**: This is a socio-politically motivated global search strategy, which is based on the imperialistic competition.
+
+- **Linear Discriminant Analysis**: This is a method used in statistics, pattern recognition, and machine learning to find a linear combination of features that characterizes or separates two or more classes of objects or events.
+
+- **Particle Filter**: This is a statistical filter technique used to estimate the state of a system where the state model and the measurements are both nonlinear.
+
+- **Particle Swarm Optimization**: This is a computational method that optimizes a problem by iteratively trying to improve a candidate solution with regard to a given measure of quality, mimicking the social behavior of bird flocking or fish schooling.
+
+- **Parzen Tree Estimator**: This is a non-parametric method to estimate the density function of random variables.
+
+- **Shuffled Frog Leaping Algorithm**: This is a metaheuristic optimization algorithm inspired by the memetic evolution of a group of frogs when searching for food.
+
+- **Simulated Annealing**: This is a probabilistic technique for approximating the global optimum of a given function, mimicking the process of heating a material and then slowly lowering the temperature to decrease defects, thus minimizing the system energy.
+
+- **Sine Cosine Algorithm**: This is a new population-based meta-heuristic algorithm, inspired by the mathematical sine and cosine functions.
+
+- **Squirrel Search Algorithm**: This is a new nature-inspired metaheuristic optimization algorithm, inspired by the caching behavior of squirrels.
+
+- **Stochastic Diffusion Search**: This is a population-based search algorithm, based on the behavior of ants when searching for food.
+
+- **Stochastic Fractal Search**: This is a metaheuristic search algorithm inspired by the natural phenomenon of fractal shapes and Brownian motion.
+
+- **Successive Linear Programming**: This is an optimization method for nonlinear optimization problems.
+
+- **Tabu Search**: This is a metaheuristic search method employing local search methods used for mathematical optimization.
+
+- **Variable Depth Search**: This is a search algorithm that explores the search space by variable-depth first search and backtracking.
+
+- **Variable Neighbourhood Search**: This is a metaheuristic search method for discrete optimization problems.
+
+- **Very Large Scale Neighborhood Search**: This is a search method that explores very large neighborhoods with an efficient algorithm.
+
+- **Whale Optimization Algorithm**: This is a new bio-inspired optimization algorithm, which simulates the social behavior of humpback whales.
+
+> [!NOTE]
+> Please note that not all of these algorithms are suitable for all types of optimization problems. Some are better suited for continuous optimization problems, some for discrete optimization problems, and others for specific types of problems like quadratic programming or linear discriminant analysis.
+
+## Contributing
+
+Contributions to Useful Optimizer are welcome! Please read the contributing guidelines before getting started.
+
+## [License](LICENSE)
+
+---
+
+> [!WARNING]
+> This project was generated with GitHub Copilot and may not be completely verified. Please use with caution and feel free to report any issues you encounter. Thank you!
+
+> [!WARNING]
+> Some parts still contain the _legacy_ `np.random.rand` call. See also: https://docs.astral.sh/ruff/rules/numpy-legacy-random/
+

useful_optimizer-0.1.1/README.md

@@ -0,0 +1,144 @@
+# Useful Optimizer
+
+Useful Optimizer is a dedicated set of optimization algorithms for numeric problems. It's designed to provide a comprehensive collection of optimization techniques that can be easily used and integrated into any project.
+
+## Version
+
+The current version of Useful Optimizer is 0.1.0.
+
+## Features
+
+- A wide range of optimization algorithms.
+- Easy to use and integrate.
+- Suitable for various numeric problems.
+- Having fun to play with the algorithms
+
+## Installation
+
+To install Useful Optimizer, you can use pip:
+
+```bash
+pip install git+https://github.com/Anselmoo/useful-optimizer
+```
+
+## Usage
+
+Here's a basic example of how to use Useful Optimizer:
+
+```python
+from opt.benchmark import CrossEntropyMethod
+from opt.benchmark.functions import shifted_ackley
+
+optimizer = CrossEntropyMethod(
+        func=shifted_ackley,
+        dim=2,
+        lower_bound=-12.768,
+        upper_bound=+12.768,
+        population_size=100,
+        max_iter=1000,
+    )
+best_solution, best_fitness = optimizer.search()
+print(f"Best solution: {best_solution}")
+print(f"Best fitness: {best_fitness}")
+```
+
+## Current Implemented Optimizer
+
+The current version of Useful Optimizer includes a wide range of optimization algorithms, each implemented as a separate module. Here's a brief overview of the implemented optimizers:
+
+Sure, here's a brief description of each optimizer:
+
+- **Adadelta, Adagrad, and Adaptive Moment Estimation**: These are gradient-based optimization algorithms commonly used in machine learning and deep learning.
+
+- **Ant Colony Optimization**: This is a nature-inspired algorithm that mimics the behavior of ants to solve optimization problems.
+
+- **Artificial Fish Swarm Algorithm**: This algorithm simulates the behavior of fish in nature to perform global optimization.
+
+- **Augmented Lagrangian Method**: This is a method to solve constrained optimization problems. It combines the objective function and the constraints into a single function using Lagrange multipliers.
+
+- **Bat Algorithm**: This is a metaheuristic optimization algorithm inspired by the echolocation behavior of microbats.
+
+- **Bee Algorithm**: This is a population-based search algorithm inspired by the food foraging behavior of honey bee colonies.
+
+- **Cat Swarm Optimization**: This algorithm is based on the behavior of cats and distinguishes between two forms of behavior in cats: seeking mode and tracing mode.
+
+- **CMA-ES (Covariance Matrix Adaptation Evolution Strategy)**: This is an evolutionary algorithm for difficult non-linear non-convex optimization problems in continuous domain.
+
+- **Colliding Bodies Optimization**: This is a physics-inspired optimization method, based on the collision and explosion of bodies.
+
+- **Cross Entropy Method**: This is a Monte Carlo method for importance sampling and optimization.
+
+- **Cuckoo Search**: This is a nature-inspired metaheuristic optimization algorithm, which is based on the obligate brood parasitism of some cuckoo species.
+
+- **Cultural Algorithm**: This is a type of evolutionary algorithm that is based on the concept of culture, or shared information and knowledge.
+
+- **Differential Evolution**: This is a population-based metaheuristic algorithm, which uses mechanisms inspired by biological evolution, such as reproduction, mutation, recombination, and selection.
+
+- **Eagle Strategy**: This is a metaheuristic optimization algorithm inspired by the hunting behavior of eagles.
+
+- **Estimation of Distribution Algorithm**: This is a stochastic optimization algorithm that uses a probabilistic model of candidate solutions.
+
+- **Firefly Algorithm**: This is a nature-inspired metaheuristic optimization algorithm, based on the flashing behavior of fireflies.
+
+- **Genetic Algorithm**: This is a search heuristic that is inspired by Charles Darwin’s theory of natural evolution.
+
+- **Glowworm Swarm Optimization**: This is a nature-inspired optimization algorithm based on the behavior of glowworms.
+
+- **Grey Wolf Optimizer**: This is a metaheuristic optimization algorithm inspired by grey wolves.
+
+- **Harmony Search**: This is a music-inspired metaheuristic optimization algorithm.
+
+- **Hessian Free Optimization**: This is a second-order optimization algorithm that uses information from the Hessian matrix to guide the search.
+
+- **Hill Climbing**: This is a mathematical optimization technique which belongs to the family of local search.
+
+- **Imperialist Competitive Algorithm**: This is a socio-politically motivated global search strategy, which is based on the imperialistic competition.
+
+- **Linear Discriminant Analysis**: This is a method used in statistics, pattern recognition, and machine learning to find a linear combination of features that characterizes or separates two or more classes of objects or events.
+
+- **Particle Filter**: This is a statistical filter technique used to estimate the state of a system where the state model and the measurements are both nonlinear.
+
+- **Particle Swarm Optimization**: This is a computational method that optimizes a problem by iteratively trying to improve a candidate solution with regard to a given measure of quality, mimicking the social behavior of bird flocking or fish schooling.
+
+- **Parzen Tree Estimator**: This is a non-parametric method to estimate the density function of random variables.
+
+- **Shuffled Frog Leaping Algorithm**: This is a metaheuristic optimization algorithm inspired by the memetic evolution of a group of frogs when searching for food.
+
+- **Simulated Annealing**: This is a probabilistic technique for approximating the global optimum of a given function, mimicking the process of heating a material and then slowly lowering the temperature to decrease defects, thus minimizing the system energy.
+
+- **Sine Cosine Algorithm**: This is a new population-based meta-heuristic algorithm, inspired by the mathematical sine and cosine functions.
+
+- **Squirrel Search Algorithm**: This is a new nature-inspired metaheuristic optimization algorithm, inspired by the caching behavior of squirrels.
+
+- **Stochastic Diffusion Search**: This is a population-based search algorithm, based on the behavior of ants when searching for food.
+
+- **Stochastic Fractal Search**: This is a metaheuristic search algorithm inspired by the natural phenomenon of fractal shapes and Brownian motion.
+
+- **Successive Linear Programming**: This is an optimization method for nonlinear optimization problems.
+
+- **Tabu Search**: This is a metaheuristic search method employing local search methods used for mathematical optimization.
+
+- **Variable Depth Search**: This is a search algorithm that explores the search space by variable-depth first search and backtracking.
+
+- **Variable Neighbourhood Search**: This is a metaheuristic search method for discrete optimization problems.
+
+- **Very Large Scale Neighborhood Search**: This is a search method that explores very large neighborhoods with an efficient algorithm.
+
+- **Whale Optimization Algorithm**: This is a new bio-inspired optimization algorithm, which simulates the social behavior of humpback whales.
+
+> [!NOTE]
+> Please note that not all of these algorithms are suitable for all types of optimization problems. Some are better suited for continuous optimization problems, some for discrete optimization problems, and others for specific types of problems like quadratic programming or linear discriminant analysis.
+
+## Contributing
+
+Contributions to Useful Optimizer are welcome! Please read the contributing guidelines before getting started.
+
+## [License](LICENSE)
+
+---
+
+> [!WARNING]
+> This project was generated with GitHub Copilot and may not be completely verified. Please use with caution and feel free to report any issues you encounter. Thank you!
+
+> [!WARNING]
+> Some parts still contain the _legacy_ `np.random.rand` call. See also: https://docs.astral.sh/ruff/rules/numpy-legacy-random/

useful_optimizer-0.1.1/opt/__init__.py

@@ -0,0 +1,6 @@
+"""Useful optimizers, a set of optimization algorithms."""
+
+from __future__ import annotations
+
+
+__version__ = "0.1.1"

useful_optimizer-0.1.1/opt/abstract_optimizer.py

@@ -0,0 +1,74 @@
+"""This module defines an abstract base class for optimizers."""
+
+from __future__ import annotations
+
+from abc import ABC
+from abc import abstractmethod
+from typing import TYPE_CHECKING
+
+import numpy as np
+
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
+    from numpy import ndarray
+
+
+class AbstractOptimizer(ABC):
+    """An abstract base class for optimizers.
+
+    Args:
+        func (Callable): The objective function to be optimized.
+        lower_bound (float): The lower bound of the search space.
+        upper_bound (float): The upper bound of the search space.
+        dim (int): The dimensionality of the search space.
+        max_iter (int, optional): The maximum number of iterations for the optimization process. Defaults to 1000.
+
+    Attributes:
+        func (Callable): The objective function to be optimized.
+        lower_bound (float): The lower bound of the search space.
+        upper_bound (float): The upper bound of the search space.
+        dim (int): The dimensionality of the search space.
+        max_iter (int): The maximum number of iterations for the optimization process.
+        seed (int): The seed for the random number generator.
+        population_size (int): The number of individuals in the population.
+
+
+    Methods:
+        search() -> Tuple[TBestSolution, TBestFitness]:
+            Perform the optimization search.
+
+    Returns:
+        Tuple containing the best solution found and its corresponding fitness value.
+    """
+
+    def __init__(
+        self,
+        func: Callable[[ndarray], float],
+        lower_bound: float,
+        upper_bound: float,
+        dim: int,
+        max_iter: int = 1000,
+        seed: int | None = None,
+        population_size: int = 100,
+    ) -> None:
+        """Initialize the optimizer."""
+        self.func = func
+        self.lower_bound = lower_bound
+        self.upper_bound = upper_bound
+        self.dim = dim
+        self.max_iter = max_iter
+        if seed is None:
+            self.seed = np.random.default_rng(42).integers(0, 2**32)
+        else:
+            self.seed = seed
+        self.population_size = population_size
+
+    @abstractmethod
+    def search(self) -> tuple[np.ndarray, float]:
+        """Perform the optimization search.
+
+        Returns:
+            Tuple containing the best solution found and its corresponding fitness value.
+        """

useful_optimizer-0.1.1/opt/adadelta.py

@@ -0,0 +1,148 @@
+"""AdaDelta Optimizer.
+
+This module implements the AdaDelta optimizer, which is an extension of AdaGrad that
+seeks to reduce its sensitivity to the learning rate hyperparameter.
+
+AdaDelta is a gradient-based optimization algorithm that adapts the learning rate
+for each of the parameters in the model. It is designed to converge faster than
+AdaGrad by using a moving average of the squared gradient values to scale the learning rate.
+
+The AdaDelta optimizer is defined by the following update rule:
+
+    Eg = rho * Eg + (1 - rho) * g^2
+    dx = -sqrt(Edx + eps) / sqrt(Eg + eps) * g
+    Edx = rho * Edx + (1 - rho) * dx^2
+    x = x + dx
+
+where:
+    - x: current solution
+    - g: gradient of the objective function
+    - rho: decay rate
+    - eps: small constant to avoid dividing by zero
+    - Eg: moving average of squared gradient values
+    - Edx: moving average of squared updates
+
+The algorithm iteratively updates the solution x by computing the gradient of the
+objective function at x, scaling it by the moving average of the squared gradients,
+and dividing it by the square root of the moving average of the squared updates.
+
+The algorithm continues for a fixed number of iterations or until a specified
+stopping criterion is met, returning the best solution found.
+
+This module provides a simple example of how to use the AdaDelta optimizer to minimize
+the Shifted Ackley's function in two dimensions.
+
+Example:
+    $ python adadelta.py
+
+Attributes:
+    rho (float): Decay rate for the moving average of squared gradients
+    eps (float): Small constant to avoid dividing by zero
+    Eg (np.ndarray): Moving average of squared gradient values
+    Edx (np.ndarray): Moving average of squared updates
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import numpy as np
+
+from scipy.optimize import approx_fprime
+
+from opt.abstract_optimizer import AbstractOptimizer
+from opt.benchmark.functions import shifted_ackley
+
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
+    from numpy import ndarray
+
+
+class AdaDelta(AbstractOptimizer):
+    """AdaDelta optimizer implementation.
+
+    Args:
+        func (Callable[[ndarray], float]): The objective function to be minimized.
+        lower_bound (float): The lower bound of the search space.
+        upper_bound (float): The upper bound of the search space.
+        dim (int): The dimensionality of the search space.
+        max_iter (int, optional): The maximum number of iterations. Defaults to 1000.
+        seed (Optional[int], optional): The seed value for random number generation. Defaults to None.
+        rho (float, optional): The decay rate for the moving average of squared gradients. Defaults to 0.97.
+        eps (float, optional): A small constant to avoid division by zero. Defaults to 1e-8.
+    """
+
+    def __init__(
+        self,
+        func: Callable[[ndarray], float],
+        lower_bound: float,
+        upper_bound: float,
+        dim: int,
+        max_iter: int = 1000,
+        rho: float = 0.97,
+        eps: float = 1e-8,
+        seed: int | None = None,
+    ) -> None:
+        """Initialize the AdaDelta optimizer."""
+        super().__init__(
+            func=func,
+            lower_bound=lower_bound,
+            upper_bound=upper_bound,
+            dim=dim,
+            max_iter=max_iter,
+            seed=seed,
+        )
+        self.rho = rho
+        self.eps = eps
+        self.Eg = np.zeros(self.dim)
+        self.Edx = np.zeros(self.dim)
+
+    def _update(self, x: np.ndarray) -> np.ndarray:
+        """Perform a single update step of the AdaDelta algorithm.
+
+        Args:
+            x (np.ndarray): The current solution.
+
+        Returns:
+            np.ndarray: The updated solution.
+        """
+        g = approx_fprime(x, self.func, self.eps)
+        self.Eg = self.rho * self.Eg + (1 - self.rho) * g**2
+        dx = -np.sqrt(self.Edx + self.eps) / np.sqrt(self.Eg + self.eps) * g
+        self.Edx = self.rho * self.Edx + (1 - self.rho) * dx**2
+        return x + dx
+
+    def search(self) -> tuple[np.ndarray, float]:
+        """Perform the AdaDelta search.
+
+        Returns:
+            Tuple[np.ndarray, float]: A tuple containing the best solution found and its fitness value.
+        """
+        # Initialize solution
+        x = np.random.default_rng(self.seed).uniform(
+            self.lower_bound, self.upper_bound, self.dim
+        )
+        best_solution = x
+        best_fitness = np.inf
+
+        # Main loop
+        for _ in range(self.max_iter):
+            x = self._update(x)
+            x = np.clip(x, self.lower_bound, self.upper_bound)
+            fitness = self.func(x)
+            if fitness < best_fitness:
+                best_fitness = fitness
+                best_solution = x
+
+        return best_solution, best_fitness
+
+
+if __name__ == "__main__":
+    optimizer = AdaDelta(
+        func=shifted_ackley, lower_bound=-2.768, upper_bound=+2.768, dim=2
+    )
+    best_solution, best_fitness = optimizer.search()
+    print(f"Best solution found: {best_solution}")
+    print(f"Best fitness value: {best_fitness}")

useful_optimizer-0.1.1/opt/adagrad.py

@@ -0,0 +1,111 @@
+"""ADAGrad Optimizer.
+
+This module implements the Adaptive Gradient Algorithm (ADAGrad) optimizer. ADAGrad is
+a gradient-based optimization algorithm that adapts the learning rate to the parameters,
+performing smaller updates for parameters associated with frequently occurring features,
+and larger updates for parameters associated with infrequent features. It is particularly
+useful for dealing with sparse data.
+
+ADAGrad's main strength is that it eliminates the need to manually tune the learning rate.
+Most implementations also include a 'smoothing term' to avoid division by zero when the
+gradient is zero.
+
+The ADAGrad optimizer is commonly used in machine learning and deep learning applications.
+
+Example:
+    optimizer = ADAGrad(func=objective_function, learning_rate=0.01, initial_guess=[0, 0])
+    best_solution, best_fitness = optimizer.optimize()
+
+Attributes:
+    func (Callable): The objective function to optimize.
+    learning_rate (float): The learning rate for the optimization.
+    initial_guess (List[float]): The starting point for the optimization.
+
+Methods:
+    optimize(): Perform the ADAGrad optimization.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import numpy as np
+
+from scipy.optimize import approx_fprime
+
+from opt.abstract_optimizer import AbstractOptimizer
+from opt.benchmark.functions import shifted_ackley
+
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
+    from numpy import ndarray
+
+
+class ADAGrad(AbstractOptimizer):
+    """ADAGrad optimizer implementation.
+
+    Args:
+        func (Callable[[ndarray], float]): The objective function to be optimized.
+        lower_bound (float): The lower bound of the search space.
+        upper_bound (float): The upper bound of the search space.
+        dim (int): The dimensionality of the search space.
+        max_iter (int, optional): The maximum number of iterations. Defaults to 1000.
+        seed (Optional[int], optional): The seed value for random number generation. Defaults to None.
+        lr (float, optional): The learning rate. Defaults to 0.01.
+        eps (float, optional): A small value added to the denominator for numerical stability. Defaults to 1e-8.
+    """
+
+    def __init__(
+        self,
+        func: Callable[[ndarray], float],
+        lower_bound: float,
+        upper_bound: float,
+        dim: int,
+        max_iter: int = 1000,
+        lr: float = 0.01,
+        eps: float = 1e-8,
+        seed: int | None = None,
+    ) -> None:
+        """Initialize the ADAGrad optimizer."""
+        super().__init__(
+            func=func,
+            lower_bound=lower_bound,
+            upper_bound=upper_bound,
+            dim=dim,
+            max_iter=max_iter,
+            seed=seed,
+        )
+        self.lr = lr
+        self.eps = eps
+
+    def search(self) -> tuple[np.ndarray, float]:
+        """Perform the ADAGrad search algorithm.
+
+        Returns:
+            Tuple[np.ndarray, float]: A tuple containing the best solution found and its fitness value.
+        """
+        x = np.random.default_rng(self.seed).uniform(
+            self.lower_bound, self.upper_bound, self.dim
+        )
+        grad_accumulator = np.zeros(self.dim)
+
+        for _ in range(self.max_iter):
+            grad = approx_fprime(x, self.func, np.sqrt(np.finfo(float).eps))
+            grad_accumulator += grad**2
+            adjusted_grad = grad / (np.sqrt(grad_accumulator) + self.eps)
+            x = x - self.lr * adjusted_grad
+
+        best_solution = x
+        best_fitness = self.func(best_solution)
+        return best_solution, best_fitness
+
+
+if __name__ == "__main__":
+    optimizer = ADAGrad(
+        func=shifted_ackley, lower_bound=-2.768, upper_bound=2.768, dim=2
+    )
+    best_solution, best_fitness = optimizer.search()
+    print(f"Best solution found: {best_solution}")
+    print(f"Best fitness value: {best_fitness}")