diffinytrace 2.1__py3-none-any.whl

diffinytrace/optimize.py

@@ -0,0 +1,808 @@
+r"""
+Optimization Utilities for PyTorch-SciPy Integration
+====================================================
+
+This submodule provides a set of tools for constrained and unconstrained optimization of PyTorch models using SciPy optimizers. It bridges the gap between SciPy’s powerful optimization routines and PyTorch’s autograd system, enabling flexible and efficient hybrid optimization workflows.
+
+Key Features:
+-------------
+- Seamless wrapping of PyTorch-based objective functions for use with SciPy.
+- Automatic gradient computation using PyTorch’s autograd.
+- Support for parameter bounds, including custom mask-based bounds.
+- Caching and reuse of recent function/gradient evaluations.
+- Integration with SciPy's `minimize`.
+- Optional tracking of optimization history (function values and gradient norms).
+- Utility functions for flattening/unpacking tensor parameters.
+- Conversion of PyTorch parameters to SciPy-compatible formats with bounds.
+- Support for custom constraints and callback functions.
+
+Optimization Constraints in Optical Systems
+-------------------------------------------
+
+When using optimization procedures to attain parameters of an optical system, it is important to have constraints that ensure that the optical system can be manufactured. The following demonstrates the implementation of different types of constraints in our library, with a specific focus on the positive air spacing and minimum glass thickness constraints.
+
+Constraint optimization problems can often be expressed as a *nonlinear program*, which is defined as follows (see :cite:`italiens`):
+
+.. math::
+
+    \min_{p} \quad m(p)
+
+.. math::
+
+    \text{subject to} \quad \hat{g}_i(p) \leq 0, \quad i = 1, \ldots, N_1,
+
+.. math::
+
+    \text{subject to} \quad \hat{h}_j(p) = 0, \quad j = 1, \ldots, N_2,
+
+where:
+- :math:`p \in \mathbb{R}^n` is the vector of parameters.
+- :math:`m: \mathbb{R}^n \to \mathbb{R}` is the nonlinear objective (merit) function.
+- :math:`\hat{g}_i: \mathbb{R}^n \to \mathbb{R}` are the inequality constraint functions.
+- :math:`\hat{h}_j: \mathbb{R}^n \to \mathbb{R}` are the equality constraint functions.
+
+For this type of problem, multiple numerical schemes are available in the Python library *SciPy*. Some optimization schemes also require derivative information for functions that describe constraints. For example, Sequential Least Squares Programming (SLSQP) uses the derivatives of the constraint functions :math:`\hat{g}_i` and :math:`\hat{h}_j` to find local minima.
+
+By combining the libraries PyTorch and SciPy, we leverage the strengths of two sophisticated and established libraries:
+
+1. **PyTorch**: Efficiently calculates the derivatives of the merit function :math:`m` and the constraint functions :math:`\hat{g}_i` and :math:`\hat{h}_j` using automatic differentiation. Additionally, it allows evaluation of these functions and their derivatives on a graphics card, providing significant speedups.
+2. **SciPy**: Provides well-tested traditional algorithms to find local minima. While PyTorch also has a wide variety of optimization algorithms, its main application is stochastic gradient descent in deep learning, which may not be the best choice for optimizing optical systems.
+
+Types of Constraints
+--------------------
+
+In our library, we implemented three ways to define constraints:
+
+1. **Bounds**  
+   Most numerical schemes in SciPy support bounding box constraints, allowing the definition of minimum and maximum values for each parameter. These bounds can be interpreted as constraints in the form :math:`\hat{g}_i(p) = p - C_i` or :math:`\hat{g}_i(p) = C_i - p`, where :math:`C_i \in \mathbb{R}`. This is particularly useful for distance transformations, where we can ensure that the distance parameter is never smaller than 0. For example:
+
+   >>> import diffinytrace as dit
+   >>> import torch
+   >>> dist_transform = dit.transforms.Distance(10.)
+   >>> dist_transform.distance.bounds = torch.tensor([5.0, torch.inf])
+
+   Here, **torch.inf** indicates that the distance can be arbitrarily large, with no upper bound.
+
+2. **Constant Variables**  
+   If a specific parameter should be fixed, PyTorch allows disabling gradient computation for that parameter. For example:
+
+   >>> import diffinytrace as dit
+   >>> distance_transform = dit.transforms.Distance(10.)
+   >>> distance_transform.distance.requires_grad = False
+
+   Note: While it is easy to set specific parameters as constants, it is not possible to disable gradient computation for individual parameters if the variable contains multiple values. For instance, in the case of a B-spline surface, it is not possible to disable gradient computation for individual B-spline coefficients.
+
+3. **Arbitrary Constraint Functions**  
+   Our library also supports defining nonlinear inequality constraint functions :math:`\hat{g}_i` and equality constraint functions :math:`\hat{h}_i`. Some local optimization methods require derivative information for these nonlinear constraint functions. To efficiently evaluate these derivatives, we use automatic differentiation. This is achieved by defining the constraint functions :math:`\hat{g}_i` with PyTorch and calculating their derivatives with respect to the parameters of the optical system. This approach eliminates the need for finite differences, which could significantly slow down the optimization procedure.
+"""
+
+# Copyright (c) 2025 Martin Pflaum
+# This file is part of the diffinytrace project, licensed under the MIT License.
+
+__all__ = [
+    "make_bounds_from_param",
+    "make_parameter_from_input",
+    "pack_tensors",
+    "unpack_tensors",
+    "apply_vec_to_params",
+    "set_full_if_nan",
+    "ParameterFunHelper",
+    "create_fun_and_gradient",
+    "remove_bounds",
+    "get_bounds",
+    "get_scipy_constraint",
+    "create_callback",
+    "minimize",
+    "copy_bounds_to_attr_name",
+    "set_bounds_from_params_mask"
+]
+
+import scipy
+import scipy.optimize
+from .utils.autograd import grad
+import torch
+import numpy as np
+import torch.nn as nn
+import copy
+from typing import Callable, List, Tuple, Optional
+
+def make_bounds_from_param(param):
+    """
+    Creates default bounds (-∞, ∞) for each element of the input tensor.
+
+    This function returns a tensor of shape `param.shape + [2]`, where the last
+    dimension represents the lower and upper bounds for each element in `param`.
+
+    Args:
+        param (torch.Tensor): A tensor for which bounds should be created.
+
+    Returns:
+        torch.Tensor: A tensor of shape `param.shape + [2]` where
+        `[..., 0] = -inf` (lower bounds) and `[..., 1] = inf` (upper bounds),
+        with the same dtype and device as `param`.
+    """
+    bounds = torch.zeros(list(param.shape)+[2],device=param.device,dtype=param.dtype)
+    bounds[...,0] = -torch.inf
+    bounds[...,1] = torch.inf
+    return bounds
+
+
+def make_parameter_from_input(input,bounds=None, dtype=None, device=None,bounds_attr_name="bounds"):
+    """
+    Converts input to a `torch.nn.Parameter` and attaches bounds as an attribute.
+
+    Args:
+        input (array-like or torch.Tensor): Input data.
+        bounds (torch.Tensor, optional): Bounds to attach to the parameter.
+        dtype (torch.dtype, optional): Desired tensor data type.
+        device (torch.device, optional): Device to store the parameter on.
+        bounds_attr_name (str): Attribute name used to store bounds.
+
+    Returns:
+        torch.nn.Parameter: The parameter with bounds attached as an attribute.
+    """
+    if not torch.is_tensor(input):
+        input = torch.tensor(input, dtype=dtype, device=device)
+
+    # If the input tensor has a different dtype or device, move it accordingly
+    if dtype is not None or device is not None:
+        input = input.to(device=device, dtype=dtype)
+
+    # If the input is not already a Parameter, convert it to one
+    if not isinstance(input, torch.nn.Parameter):
+        input = torch.nn.Parameter(input)
+    
+    if bounds is None:
+        bounds = make_bounds_from_param(input)
+    #input.bounds = bounds
+    setattr(input,bounds_attr_name,bounds)    
+    return input
+
+def pack_tensors(tensor_list:List[torch.Tensor]) -> torch.Tensor:
+    """
+    Flattens and concatenates a list of tensors into a single 1D tensor.
+
+    Args:
+        tensor_list (list of torch.Tensor or torch.Tensor): Input tensor(s).
+
+    Returns:
+        torch.Tensor: A 1D tensor.
+    """
+    if torch.is_tensor(tensor_list):
+        return tensor_list.reshape(-1)
+    return torch.cat([t.reshape(-1) for t in tensor_list])
+
+def unpack_tensors(packed_tensor: torch.Tensor, shapes: List[Tuple[int]]) -> List[torch.Tensor]:
+    """
+    Unpacks a 1D tensor into a list of tensors with specified shapes.
+
+    Args:
+        packed_tensor (torch.Tensor): The flat tensor to unpack.
+        shapes (list of tuple): Target shapes for unpacked tensors.
+
+    Returns:
+        list of torch.Tensor: Unpacked tensors with original shapes.
+    """
+    unpacked_tensors = []
+    start = 0
+    for shape in shapes:
+        size = torch.prod(torch.tensor(shape)).item()  # Calculate the size of the tensor
+        size = int(max(size,1))
+        # Reshape the portion of packed_tensor to the original shape
+        tensor = packed_tensor[start:start + size]
+        if len(shape) > 0:  # Only reshape if shape is not scalar
+            tensor = tensor.reshape(*shape)
+        unpacked_tensors.append(tensor)
+        start += size  # Move to the next start index
+    return unpacked_tensors
+
+def apply_vec_to_params(vec: np.ndarray, params: list[torch.nn.Parameter], device=None, dtype=None):
+    """
+    Updates PyTorch parameters with values from a flattened NumPy vector.
+
+    This function is used in optimization workflows to update parameter values
+    during SciPy optimization. It takes a flat vector of parameter values and
+    distributes them back to the original parameter tensors, preserving their
+    original shapes.
+
+    Args:
+        vec (np.ndarray): A 1D NumPy array containing new parameter values.
+            The length must match the total number of elements across all parameters.
+        params (list[torch.nn.Parameter]): List of PyTorch parameters to update.
+            Each parameter will be reshaped from the corresponding portion of `vec`.
+        device (torch.device, optional): Target device for the parameters. 
+            If None, uses the device of the first parameter. Defaults to None.
+        dtype (torch.dtype, optional): Target data type for the parameters.
+            If None, uses the dtype of the first parameter. Defaults to None.
+
+    Raises:
+        RuntimeError: If `vec` is not a NumPy array.
+
+    Example:
+        >>> import torch
+        >>> import numpy as np
+        >>> import diffinytrace as dit
+        >>> 
+        >>> # Create some parameters
+        >>> params = [
+        ...     torch.nn.Parameter(torch.ones((2,2)))*0.25,
+        ...     torch.nn.Parameter(torch.ones(3))
+        ... ]
+        >>> # Flatten parameters to create a vector
+        >>> vec = dit.optimize.pack_tensors(params).detach().cpu().numpy()
+        >>> print(f"Vector length: {len(vec)}")  # Should be 2*2 + 3 = 7
+        >>> # Modify the vector
+        >>> 
+        >>> print(params)
+        >>> 
+        >>> vec_new = vec * 2.0
+        >>> # Update parameters with new values
+        >>> dit.optimize.apply_vec_to_params(vec_new, params)
+        >>> 
+        >>> # Parameters are now updated with doubled values
+        >>> print(params)
+
+    Note:
+        - This function modifies parameters in-place using `param.data = ...`
+        - The function uses `torch.no_grad()` to avoid building computation graphs
+        - Parameter shapes are preserved during the update process
+        - Commonly used with `pack_tensors()` and `unpack_tensors()` for optimization
+    """
+    if not isinstance(vec, np.ndarray):
+        raise RuntimeError("vec should be a numpy vector")
+    params = [elem for elem in params]
+    if dtype is None:
+        dtype = params[0].dtype
+    if device is None:
+        device = params[0].device
+    unpacked_params = unpack_tensors(torch.tensor(vec,device=device,dtype=dtype), [elem.shape for elem in params])
+    with torch.no_grad():
+        for k,param in enumerate(params):
+            param.data = unpacked_params[k]
+
+def set_full_if_nan(input:np.ndarray, fill_value: float)->np.ndarray:
+    """
+    Replaces NaNs in input with a specified fill value.
+
+    Args:
+        input (np.ndarray): A NumPy array or scalar.
+        fill_value (float): Value to use in place of NaNs.
+
+    Returns:
+        np.ndarray or float: Modified input with no NaNs.
+    """
+    if not isinstance(input, np.ndarray):
+        raise RuntimeError("set_full_if_nan,input should be a numpy vector")
+    
+    if len(input.shape) == 0:
+        if np.isnan(input):
+            return np.array(fill_value)
+        else:
+            return input
+    else:
+        if np.isnan(input).any():
+            input = np.full_like(input, fill_value)
+            return input
+        else:
+            return input
+
+class ParameterFunHelper():
+    """
+    Helper class for evaluating PyTorch functions and gradients in SciPy optimization.
+    
+    This class bridges PyTorch's automatic differentiation with SciPy's optimization
+    routines by providing function and gradient evaluations in NumPy format.
+    It includes caching to avoid redundant computations and handles NaN values
+    gracefully during optimization.
+
+    Args:
+        original_fun (Callable): PyTorch function to be optimized. Should return a scalar tensor.
+        params (List[torch.nn.Parameter]): List of PyTorch parameters to optimize over.
+        nan_fallback (float, optional): Value to return if NaN is detected in function 
+            or gradient evaluation. Defaults to float("inf").
+
+    Attributes:
+        original_fun (Callable): The objective function being optimized.
+        params (List[torch.nn.Parameter]): Parameters for optimization.
+        nan_fallback (float): Fallback value for NaN handling.
+        last_x_fun_numpy (np.ndarray): Cache of last input for function evaluation.
+        last_fun_val_numpy (float): Cache of last function value in NumPy format.
+        last_fun_val_torch (torch.Tensor): Cache of last function value as PyTorch tensor.
+        last_x_grad_numpy (np.ndarray): Cache of last input for gradient evaluation.
+        last_grad_val_numpy (np.ndarray): Cache of last gradient in NumPy format.
+
+    Example:
+        >>> import torch
+        >>> import diffinytrace as dit
+        >>> import numpy as np
+        >>> 
+        >>> # Define parameters and objective function
+        >>> params = [torch.nn.Parameter(torch.randn(5))]
+        >>> def objective():
+        ...     return torch.sum(params[0]**2)
+        >>> 
+        >>> # Create helper for SciPy optimization
+        >>> helper = dit.optimize.ParameterFunHelper(objective, params)
+        >>> 
+        >>> # Use with SciPy
+        >>> x0 = np.ones((5,))*3.
+        >>> fun_val = helper.fun(x0)        # Evaluate function 5*3^2 = 45
+        >>> grad_val = helper.jac(x0)       # Evaluate gradient 2*3 = 6
+        >>> fun_val, grad_val = helper.fun_jac(x0)  # Evaluate both
+        >>> 
+        >>> print(fun_val, grad_val)  # (45.0, array([6., 6., 6., 6., 6.]))
+
+    Note:
+        - Function and gradient evaluations are cached to avoid redundant computations
+          when SciPy requests the same point multiple times.
+        - All NaN values in function outputs or gradients are replaced with `nan_fallback`.
+        - Parameters are automatically updated with new values during evaluation.
+    """
+    def __init__(self,orginal_fun,params,nan_fallback = float("inf")):
+        self.last_x_fun_numpy = None
+        self.last_fun_val_numpy = None
+        self.last_fun_val_torch = None
+        
+        self.last_x_grad_numpy = None
+        self.last_grad_val_numpy = None
+        self.orginal_fun = orginal_fun
+
+        self.params = [param for param in params]
+        self.nan_fallback = nan_fallback
+        
+    def fun(self,x):
+        """
+        Evaluates the objective function at a given input.
+
+        Args:
+            x (np.ndarray): Flat input array.
+
+        Returns:
+            float: Function value with NaNs replaced if needed.
+        """
+        if not self.last_x_fun_numpy is None:
+            if (x == self.last_x_fun_numpy).all():
+                out = self.last_fun_val_numpy
+                out = set_full_if_nan(out,self.nan_fallback)
+                return out
+        
+        
+        device = self.params[0].device
+        dtype = self.params[0].dtype
+        apply_vec_to_params(x,self.params,device,dtype)    
+        self.last_x_fun_numpy = copy.deepcopy(x)
+        fun_val = self.orginal_fun()
+        self.last_fun_val_torch = fun_val
+        self.last_fun_val_numpy = set_full_if_nan(fun_val.detach().cpu().numpy(),self.nan_fallback)
+        out = self.last_fun_val_numpy
+        out = set_full_if_nan(out,self.nan_fallback)
+        return out
+     
+    def jac(self,x):
+        """
+        Computes the gradient of the objective function at input x.
+
+        Args:
+            x (np.ndarray): Flat input array.
+
+        Returns:
+            np.ndarray: Gradient with NaNs replaced if needed.
+        """
+        if not self.last_x_grad_numpy is None:
+            if (x == self.last_x_grad_numpy).all():
+                out = self.last_grad_val_numpy
+                out = set_full_if_nan(out,self.nan_fallback)
+                return out
+        
+        self.fun(x)
+        self.last_x_grad_numpy = copy.deepcopy(x)
+        dp = grad(self.last_fun_val_torch,inputs=self.params,materialize_grads=True,create_graph=False,retain_graph=False) 
+        dp = pack_tensors(dp)
+        dp_numpy = dp.detach().cpu().numpy()
+        
+        self.last_grad_val_numpy = set_full_if_nan(dp_numpy,self.nan_fallback)
+
+        out = dp_numpy
+        out = set_full_if_nan(out,self.nan_fallback)
+        return out
+            
+    def fun_jac(self,x):
+        """
+        Evaluates both function value and gradient at once.
+
+        Args:
+            x (np.ndarray): Flat input array.
+
+        Returns:
+            Tuple[float, np.ndarray]: Function value and gradient.
+        """
+        fun_val_numpy = self.fun(x)
+        grad_val_numpy = self.jac(x)
+        return fun_val_numpy,grad_val_numpy
+    """
+    def hess(self,x,v):
+        if not self.calc_hess:
+            raise("ParameterFunHelper: calc_hess was initialized with False!")
+        device = self.last_grad_val_torch.device
+        dtype = self.last_grad_val_torch.dtype
+        self.grad(x)
+        v_torch = torch.tensor(v,device=device,dtype=dtype)
+        Hv = grad(self.last_grad_val_torch,inputs=self.params,grad_outputs=v_torch,materialize_grads=True,create_graph=False,retain_graph=True)
+        Hv_packed = pack_tensors(Hv)
+        out = Hv_packed.detach().cpu().numpy()
+        out = set_full_if_nan(out,self.nan_fallback)
+        print("hess out ",out)
+        return out"""
+
+ 
+def create_fun_and_gradient(merit_fun,params,nan_fallback,device,dtype):
+    """
+    Wraps a PyTorch merit function and returns a callable that evaluates both
+    the function and its gradient in NumPy format.
+
+    Args:
+        merit_fun (Callable): PyTorch function to optimize.
+        params (list): List of `torch.nn.Parameter` objects.
+        nan_fallback (float): Value to use if NaNs are encountered.
+        device (torch.device): Target device.
+        dtype (torch.dtype): Target dtype.
+
+    Returns:
+        Callable: Function that returns (value, gradient) as NumPy arrays.
+    """
+    def fun_and_gradient(input):
+        apply_vec_to_params(input,params,device,dtype)    
+        merit_val = merit_fun()
+        dmdp = grad(merit_val,inputs=params,materialize_grads=True,create_graph=False,retain_graph=False)
+        
+        out_merit_val = merit_val.detach().cpu()
+        out_dmdp = [elem.detach().cpu() for elem in dmdp]
+        out_dmdp = pack_tensors(out_dmdp)
+        
+        out_dmdp = set_full_if_nan(out_dmdp.numpy(),nan_fallback)
+        out_merit_val = set_full_if_nan(out_merit_val.numpy(),nan_fallback)
+        
+        #print("merit_val: ",out_merit_val)
+        return out_merit_val,out_dmdp
+    return fun_and_gradient
+
+
+def remove_bounds(params,bounds_attr_name) -> None:
+    """
+    Removes the bounds attribute from parameters if present.
+
+    Args:
+        params (list): List of torch.nn.Parameter objects.
+        bounds_attr_name (str): Attribute name of bounds to remove.
+    """
+    for elem in params:
+        if hasattr(elem,bounds_attr_name):
+            setattr(elem,bounds_attr_name,None)
+
+def get_bounds(params,bounds_attr_name="bounds"):
+    """
+    Extracts and concatenates bounds for all parameters.
+
+    Args:
+        params (list): List of torch.nn.Parameter objects.
+        bounds_attr_name (str): Name of attribute storing bounds.
+
+    Returns:
+        np.ndarray: Array of shape (N, 2) with all bounds.
+    """
+    out = []
+
+    for elem in params:
+        if not hasattr(elem,bounds_attr_name):
+            bounds = make_bounds_from_param(elem) 
+            setattr(elem,bounds_attr_name,bounds)
+        tmp = getattr(elem,bounds_attr_name)
+        if isinstance(tmp,list):
+            tmp = torch.tensor(np.array(tmp),dtype=torch.get_default_dtype())
+        if isinstance(tmp,np.ndarray):
+            tmp = torch.tensor(tmp,dtype=torch.get_default_dtype())
+        
+        out += [tmp]
+    out = torch.cat([t.reshape(-1,2) for t in out],dim=0)
+    out = out.detach().cpu()
+    #print("out",out)
+    out = np.array(out)
+    return out
+    
+def get_scipy_constraint(constraint,params,nan_fallback):
+    """
+    Converts a constraint into SciPy-compatible format.
+
+    Args:
+        constraint (Constraint): A custom constraint object.
+        params (list): List of parameters for the optimization.
+        nan_fallback (float): Fallback value for NaNs.
+
+    Returns:
+        dict: A dictionary compatible with SciPy constraints.
+    """
+    param_fun_helper = ParameterFunHelper(constraint.fun,params,nan_fallback)
+    param_fun_helper.constraint=True
+
+    scipy_data = {'type': constraint.type,'fun':param_fun_helper.fun,'jac':param_fun_helper.jac}
+    return scipy_data
+
+
+def create_callback(callback_fun,params,device,dtype):
+    """
+    Wraps a PyTorch callback function for use in SciPy.
+
+    Args:
+        callback_fun (Callable): A function taking no arguments.
+        params (list): List of parameters to update before calling.
+        device (torch.device): Device of the parameters.
+        dtype (torch.dtype): Data type of the parameters.
+
+    Returns:
+        Callable: A callback function for SciPy optimizers.
+    """
+    def call_back(input):
+        apply_vec_to_params(input,params,device,dtype)    
+        return callback_fun()
+    return call_back
+
+#nlopt==2.6.2
+"""
+def global_dual_annealing(fun, 
+                          params, 
+                          constraints=[],
+                          annealing_maxiter=1000,  
+                          annealing_initial_temp=5230.0, 
+                          annealing_restart_temp_ratio=2e-05, 
+                          annealing_visit=2.62, 
+                          annealing_accept=-5.0, 
+                          annealing_maxfun=10000000.0,
+                          bounds_attr_name="bounds",
+                          local_tol=1e-6,
+                          local_method=None):
+    nan_fallback = annealing_maxfun
+    
+    from .constraints import Constraint
+
+    if isinstance(constraints,Constraint):
+        constraints = [constraints]
+
+    if local_method is None:
+        if len(constraints) == 0:
+            local_method = 'L-BFGS-B'
+        else:
+            local_method = 'SLSQP'
+
+    if (not local_method == 'SLSQP') and (len(constraints)>0):
+        raise RuntimeError("Only for method SLSQP constraints are supported!")
+    
+    if isinstance(params, torch.nn.Parameter):
+        params = [params]
+        
+    params = [param for param in params if param.requires_grad]
+
+    if len(params) == 0:
+        raise RuntimeError("Params is either an empty list or no parameter provided requires_grad!")
+
+    constraints = [get_scipy_constraint(constraint,params,nan_fallback) for constraint in constraints]    
+
+    device = params[0].device
+    dtype = params[0].dtype
+    
+    bounds_numpy = get_bounds(params,bounds_attr_name)
+    if np.isinf(bounds_numpy).any():
+        raise RuntimeError("All bounds need to be non inf!")
+    param_helper_main = ParameterFunHelper(fun,params,nan_fallback)
+    #fun_helper = ParameterFunHelper(fun,params,False,nan_fallback)
+  
+    minimizer_kwargs = dict(
+        #func=param_helper_main.fun,
+        jac=param_helper_main.jac,
+                            constraints=constraints,
+                            tol=local_tol,
+                            method=local_method)
+
+    
+
+    initial_params = pack_tensors([param.cpu().detach() for param in params])  # Pack the initial params
+    
+    result = scipy.optimize.dual_annealing(
+        func=param_helper_main.fun,
+        x0=initial_params, 
+        bounds=bounds_numpy,
+        maxiter = annealing_maxiter,  
+        initial_temp=annealing_initial_temp, 
+        restart_temp_ratio = annealing_restart_temp_ratio, 
+        visit = annealing_visit, 
+        accept = annealing_accept, 
+        maxfun=annealing_maxfun,
+        minimizer_kwargs=minimizer_kwargs)
+    
+
+    apply_vec_to_params(result["x"],[p for p in params],device,dtype)    
+    return result
+"""
+
+
+def minimize(fun, 
+             params, 
+             constraints:List=[], 
+             method=None, 
+             tol:float=1e-9,
+             callback:Callable=lambda:None, 
+             options:Optional[dict]=None,
+             nan_fallback:float=float("inf"),
+             bounds_attr_name:str="bounds",
+             save_history:bool=False,
+             call_before_minimize:bool=False)->dict:
+    """
+    Minimizes a function using SciPy's `minimize`, supporting bounds and constraints.
+
+    Args:
+        fun (Callable): Objective function.
+        params (list): Parameters to optimize.
+        constraints (list): List of constraints.
+        method (str): SciPy optimization method (e.g., 'L-BFGS-B').
+        tol (float): Tolerance for convergence.
+        callback (Callable): Optional callback function.
+        options (dict): Optimizer options.
+        nan_fallback (float): Value to use if function returns NaN.
+        bounds_attr_name (str): Name of bounds attribute.
+        save_history (bool): If True, saves function values and gradient norms.
+        call_before_minimize (bool): Whether to evaluate once before optimization.
+
+    Returns:
+        dict: Dictionary containing optimization results (and optionally history).
+    """
+    from .constraints import Constraint
+
+    if isinstance(constraints,Constraint):
+        constraints = [constraints]
+
+    if method is None:
+        if len(constraints) == 0:
+            method = 'L-BFGS-B'
+        else:
+            method = 'SLSQP'
+
+    if (not method == 'SLSQP') and (len(constraints)>0):
+        raise RuntimeError("Only for method SLSQP constraints are supported!")
+    
+    if isinstance(params, torch.nn.Parameter):
+        params = [params]
+        
+    params = [param for param in params if param.requires_grad]
+
+    if len(params) == 0:
+        raise RuntimeError("Params is either an empty list or no parameter provided requires_grad!")
+
+    constraints = [get_scipy_constraint(constraint,params,nan_fallback) for constraint in constraints]    
+
+    device = params[0].device
+    dtype = params[0].dtype
+    
+    bounds_numpy = get_bounds(params,bounds_attr_name)
+
+    initial_params = pack_tensors([param.cpu().detach() for param in params])  # Pack the initial params
+    param_helper_main = ParameterFunHelper(fun,params,nan_fallback)
+    #fun_helper = ParameterFunHelper(fun,params,False,nan_fallback)
+    
+    history = {"fun_vals":[],"fun_grads_norm":[]}
+
+    fun_and_gradient = param_helper_main.fun_jac#create_fun_and_gradient(fun,params,nan_fallback,device=device,dtype=dtype)
+    if save_history:
+        
+        def callback_history(input):
+            
+            out_merit_val,out_dmdp = fun_and_gradient(input)
+            history["fun_vals"] += [out_merit_val]
+            history["fun_grads_norm"] += [np.linalg.norm(out_dmdp)]
+            
+        if callback is None:
+            callback = callback_history
+        else:
+            callback_tmp = create_callback(callback,params,device,dtype)
+            def combined_callback(input):
+                callback_tmp(input)
+                callback_history(input)
+            callback = combined_callback
+    elif callback is not None:
+        callback = create_callback(callback,params,device,dtype)
+
+    initial_params = np.array(initial_params)
+    if call_before_minimize:
+        fun_and_gradient(initial_params)
+        callback(initial_params)
+    
+    result = scipy.optimize.minimize(
+            fun=fun_and_gradient,
+            x0=initial_params,
+            jac=True,  # Indicates that the function returns both value and gradient
+            bounds=bounds_numpy,
+            method=method,  # Choose an appropriate method
+            tol=tol,
+            callback=callback,
+            options=options,
+            constraints=constraints,
+            #hessp=fun_helper.hess
+        )
+    apply_vec_to_params(result["x"],[p for p in params],device,dtype)    
+    result = {key:result[key] for key in result.keys()}
+    if len(history["fun_vals"])>0:
+        history["fun_vals"] = np.array(history["fun_vals"])
+        history["fun_grads_norm"] = np.array(history["fun_grads_norm"])
+    
+    if save_history:
+        result["history"] = history
+    return result
+
+
+def copy_bounds_to_attr_name(params,bounds_attr_name_new,bounds_attr_name_old="bounds",replace_existing_once=True):
+    """
+    Copies bounds from one attribute name to another.
+
+    Args:
+        params (list): List of parameters.
+        bounds_attr_name_new (str): New attribute name.
+        bounds_attr_name_old (str): Existing attribute name.
+        replace_existing_once (bool): Whether to skip copying if new attribute exists.
+    """
+    def copy_bounds(param,bounds_attr_name_new,bounds_attr_name_old="bounds"):
+        bounds = None
+        if hasattr(param,bounds_attr_name_old):
+            bounds = getattr(param,bounds_attr_name_old)
+        else:
+            bounds = make_bounds_from_param(param)
+        bounds = bounds.clone()
+        setattr(param,bounds_attr_name_new,bounds)
+    if isinstance(params,nn.Parameter):
+        params = [params]
+    params = [param for param in params]
+    for param in params:
+        if (not replace_existing_once) and (hasattr(param,bounds_attr_name_new)):
+            continue
+        else:
+            copy_bounds(param,bounds_attr_name_new,bounds_attr_name_old)
+
+
+def set_bounds_from_params_mask(params,mask:list|torch.Tensor,bounds_attr_name_new,bounds_attr_name_old="bounds"):
+    """
+    Sets bounds for parameters based on a mask. Parameters with `mask=False`
+    get fixed bounds (equal lower and upper bounds).
+
+    Args:
+        params (list): List of parameters.
+        mask (list or torch.Tensor): Mask specifying which elements are free.
+        bounds_attr_name_new (str): Attribute name to store new bounds.
+        bounds_attr_name_old (str): Attribute name to read old bounds from.
+    """
+    def set_new_bounds_from_param_mask(param,mask,bounds_attr_name_new,bounds_attr_name_old="bounds"):
+        bounds = None
+        if hasattr(param,bounds_attr_name_old):
+            bounds = getattr(param,bounds_attr_name_old)
+        else:
+            bounds = make_bounds_from_param(param)
+        bounds = bounds.clone()
+        bounds_shape = bounds.shape
+        mask = mask.reshape(-1)
+        bounds = bounds.reshape(-1,2)
+        data = param.data.clone()
+        data = data.reshape(-1)
+        #print("shapes",mask.shape,(mask==False).shape,bounds.shape)
+        mask_false = mask==False
+        bounds[mask_false,0] = data[mask_false]
+        bounds[mask_false,1] = data[mask_false]
+        bounds = bounds.reshape(*bounds_shape)
+        setattr(param,bounds_attr_name_new,bounds)
+
+
+    if isinstance(params,nn.Parameter):
+        params = [params]
+    params = [param for param in params]
+    if isinstance(mask,(np.ndarray)) or torch.is_tensor(mask):
+        mask = [mask]
+
+    for k in range(len(params)):
+        set_new_bounds_from_param_mask(params[k],mask[k],bounds_attr_name_new=bounds_attr_name_new,bounds_attr_name_old=bounds_attr_name_old)
+
+