DFO-LS 1.4.1__py3-none-any.whl → 1.5.0__py3-none-any.whl

{DFO_LS-1.4.1.dist-info → DFO_LS-1.5.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: DFO-LS
-Version: 1.4.1
+Version: 1.5.0
 Summary: A flexible derivative-free solver for (bound constrained) nonlinear least-squares minimization
 Author-email: Lindon Roberts <lindon.roberts@sydney.edu.au>
 Maintainer-email: Lindon Roberts <lindon.roberts@sydney.edu.au>
@@ -68,7 +68,7 @@ DFO-LS: Derivative-Free Optimizer for Least-Squares
 
 DFO-LS is a flexible package for solving nonlinear least-squares minimization, without requiring derivatives of the objective. It is particularly useful when evaluations of the objective function are expensive and/or noisy. DFO-LS is more flexible version of `DFO-GN <https://github.com/numericalalgorithmsgroup/dfogn>`_.
 
-This is an implementation of the algorithm from our paper: C. Cartis, J. Fiala, B. Marteau and L. Roberts, `Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers <https://doi.org/10.1145/3338517>`_, *ACM Transactions on Mathematical Software*, 45:3 (2019), pp. 32:1-32:41 [`preprint <https://arxiv.org/abs/1804.00154>`_]. For reproducibility of all figures in this paper, please feel free to contact the authors. 
+The main algorithm is described in our paper [1] below. 
 
 If you are interested in solving general optimization problems (without a least-squares structure), you may wish to try `Py-BOBYQA <https://github.com/numericalalgorithmsgroup/pybobyqa>`_, which has many of the same features as DFO-LS.
 
@@ -78,13 +78,15 @@ See manual.pdf or `here <https://numericalalgorithmsgroup.github.io/dfols/>`_.
 
 Citation
 --------
-If you use DFO-LS in a paper, please cite:
+The development of DFO-LS is outlined over several publications:
 
-Cartis, C., Fiala, J., Marteau, B. and Roberts, L., `Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers <https://doi.org/10.1145/3338517>`_, *ACM Transactions on Mathematical Software*, 45:3 (2019), pp. 32:1-32:41.
+1. C Cartis, J Fiala, B Marteau and L Roberts, `Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers <https://doi.org/10.1145/3338517>`_, *ACM Transactions on Mathematical Software*, 45:3 (2019), pp. 32:1-32:41 [`preprint arXiv 1804.00154 <https://arxiv.org/abs/1804.00154>`_] . 
+2. M Hough and L Roberts, `Model-Based Derivative-Free Methods for Convex-Constrained Optimization <https://doi.org/10.1137/21M1460971>`_, *SIAM Journal on Optimization*, 21:4 (2022), pp. 2552-2579 [`preprint arXiv 2111.05443 <https://arxiv.org/abs/2111.05443>`_].
+3. Y Liu, K H Lam and L Roberts, `Black-box Optimization Algorithms for Regularized Least-squares Problems <http://arxiv.org/abs/2407.14915>`_, *arXiv preprint arXiv:arXiv:2407.14915*, 2024.
 
-If you use DFO-LS for problems with constraints, including bound constraints, please also cite:
-
-Hough, M. and Roberts, L., `Model-Based Derivative-Free Methods for Convex-Constrained Optimization <https://doi.org/10.1137/21M1460971>`_, *SIAM Journal on Optimization*, 21:4 (2022), pp. 2552-2579.
+If you use DFO-LS in a paper, please cite [1]. 
+If your problem has constraints, including bound constraints, please cite [1,2].
+If your problem includes a regularizer, please cite [1,3].
 
 Requirements
 ------------
@@ -114,27 +116,13 @@ For easy installation, use `pip <http://www.pip-installer.org/>`_ as root:
 
 .. code-block:: bash
 
-    $ [sudo] pip install DFO-LS
-
-or alternatively *easy_install*:
-
-.. code-block:: bash
-
-    $ [sudo] easy_install DFO-LS
-
-If you do not have root privileges or you want to install DFO-LS for your private use, you can use:
-
-.. code-block:: bash
-
-    $ pip install --user DFO-LS
-
-which will install DFO-LS in your home directory.
+    $ pip install DFO-LS
 
 Note that if an older install of DFO-LS is present on your system you can use:
 
 .. code-block:: bash
 
-    $ [sudo] pip install --upgrade DFO-LS
+    $ pip install --upgrade DFO-LS
 
 to upgrade DFO-LS to the latest version.
 
@@ -151,22 +139,14 @@ DFO-LS is written in pure Python and requires no compilation. It can be installe
 
  .. code-block:: bash
 
-    $ [sudo] pip install .
-
-If you do not have root privileges or you want to install DFO-LS for your private use, you can use:
-
- .. code-block:: bash
-
-    $ pip install --user .
-
-instead.
+    $ pip install .
 
 To upgrade DFO-LS to the latest version, navigate to the top-level directory (i.e. the one containing :code:`pyproject.toml`) and rerun the installation using :code:`pip`, as above:
 
  .. code-block:: bash
 
     $ git pull
-    $ [sudo] pip install .  # with admin privileges
+    $ pip install .
 
 Testing
 -------
@@ -189,7 +169,7 @@ If DFO-LS was installed using *pip* you can uninstall as follows:
 
  .. code-block:: bash
 
-    $ [sudo] pip uninstall DFO-LS
+    $ pip uninstall DFO-LS
 
 If DFO-LS was installed manually you have to remove the installed files by hand (located in your python site-packages directory).
 

DFO_LS-1.5.0.dist-info/RECORD

@@ -0,0 +1,14 @@
+dfols/__init__.py,sha256=nMJ4G3JcmjQ82lYXV2ywxjHWQqd9nq7Ak6GIlrN70Tw,1605
+dfols/controller.py,sha256=gz4yGpk8KyfsWxrAkI8y69K5ckSHZ3Xdq0fEVFtIcPk,49925
+dfols/diagnostic_info.py,sha256=2kEUkL-MS4eDENUf1r2hOWsntP8OxMDKi_kyHmrC9V4,6081
+dfols/hessian.py,sha256=sExx4J4KoGwHItbthX2odosB2ONbQFvLdlcod7PIh4k,4262
+dfols/model.py,sha256=i-TcGNFAeYt4uu3R_-THTk2rOCDvgU_mcZQQXfE1ODA,19786
+dfols/params.py,sha256=GzJGO0TByH1X3B0NbLOCOqmYG8dRiKPKjjX7or_fOqI,18342
+dfols/solver.py,sha256=QUF84UYnSitvlpVssKLdcMF9e_zdA9qlZlg5e8IegeQ,63173
+dfols/trust_region.py,sha256=JbHLBDw7H88a3cIMuialh7kpMNGjL3Lp9JsjrBNpDWQ,28231
+dfols/util.py,sha256=efGVAKPb7YrHya1IOgyzacwa_h0u2jHHs5FhuxUlYDg,10282
+DFO_LS-1.5.0.dist-info/LICENSE.txt,sha256=jOtLnuWt7d5Hsx6XXB2QxzrSe2sWWh3NgMfFRetluQM,35147
+DFO_LS-1.5.0.dist-info/METADATA,sha256=JIQNs15kBtVr5_cA7JnXDbT-uQ06pqTd3RD_MRYmB7w,8069
+DFO_LS-1.5.0.dist-info/WHEEL,sha256=cVxcB9AmuTcXqmwrtPhNK88dr7IR_b6qagTj0UvIEbY,91
+DFO_LS-1.5.0.dist-info/top_level.txt,sha256=UfxRhaDN8HQx2_l17KbrDrERJ90OCN7VKkDMpYYbRLU,6
+DFO_LS-1.5.0.dist-info/RECORD,,

{DFO_LS-1.4.1.dist-info → DFO_LS-1.5.0.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: bdist_wheel (0.43.0)
+Generator: setuptools (74.1.2)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

dfols/__init__.py

@@ -39,7 +39,7 @@ alternative licensing.
 from __future__ import absolute_import, division, print_function, unicode_literals
 
 # DFO-LS version
-__version__ = '1.4.1'
+__version__ = '1.5.0'
 
 # Main solver & exit flags
 from .solver import *

dfols/controller.py

@@ -100,14 +100,19 @@ class ExitInformation(object):
 
 
 class Controller(object):
-    def __init__(self, objfun, args, x0, r0, r0_nsamples, xl, xu, projections, npt, rhobeg, rhoend, nf, nx, maxfun, params,
-                 scaling_changes, do_logging):
+    def __init__(self, objfun, argsf, x0, r0, r0_nsamples, xl, xu, projections, npt, rhobeg, rhoend, nf, nx, maxfun, params,
+                 scaling_changes, do_logging, h=None, lh=None, argsh = (), prox_uh=None, argsprox = ()):
         self.do_logging = do_logging
         self.objfun = objfun
-        self.args = args
+        self.h = h
+        self.argsf = argsf
+        self.argsh = argsh
+        self.lh = lh
+        self.prox_uh = prox_uh #TODO: add instruction for prox_uh
+        self.argsprox = argsprox
         self.maxfun = maxfun
-        self.model = Model(npt, x0, r0, xl, xu, projections, r0_nsamples, precondition=params("interpolation.precondition"),
-                           abs_tol = params("model.abs_tol"), rel_tol = params("model.rel_tol"), do_logging=do_logging)
+        self.model = Model(npt, x0, r0, xl, xu, projections, r0_nsamples, h=self.h, argsh = argsh, precondition=params("interpolation.precondition"),
+                           abs_tol = params("model.abs_tol"), rel_tol = params("model.rel_tol"), do_logging=do_logging, scaling_changes=scaling_changes)
         self.nf = nf
         self.nx = nx
         self.rhobeg = rhobeg
@@ -230,7 +235,7 @@ class Controller(object):
             for k in range(0,self.n()):
                 # Evaluate objective at this new point
                 x = self.model.as_absolute_coordinates(D[k, :])
-                rvec_list, f_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
+                rvec_list, obj_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
 
                 # Handle exit conditions (f < min obj value or maxfun reached)
                 if exit_info is not None:
@@ -289,7 +294,7 @@ class Controller(object):
 
             # Evaluate objective at this new point
             x = self.model.as_absolute_coordinates(xpts_added[k, :])
-            rvec_list, f_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
+            rvec_list, obj_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
 
             # Handle exit conditions (f < min obj value or maxfun reached)
             if exit_info is not None:
@@ -309,7 +314,7 @@ class Controller(object):
             # Note: this works because the steps for (k) and (k-n) points were in the same coordinate direction
             if self.n() + 1 <= k < 2 * self.n() + 1:
                 # Only swap if steps were in different directions AND new pt has lower objective
-                if stepa * stepb < 0.0 and self.model.fval[k] < self.model.fval[k - self.n()]:
+                if stepa * stepb < 0.0 and self.model.objval[k] < self.model.objval[k - self.n()]:
                     xpts_added[[k, k-self.n()]] = xpts_added[[k-self.n(), k]]
 
         return None   # return & continue
@@ -342,7 +347,7 @@ class Controller(object):
             for ndirns in range(num_directions):
                 new_point = xopt + dirns[ndirns, :]  # alway base move around best value so far
                 x = self.model.as_absolute_coordinates(new_point)
-                rvec_list, f_list, num_samples_run, exit_info = eval_obj_results[ndirns]
+                rvec_list, obj_list, num_samples_run, exit_info = eval_obj_results[ndirns]
                 # Handle exit conditions (f < min obj value or maxfun reached)
                 if exit_info is not None:
                     if num_samples_run > 0:
@@ -361,7 +366,7 @@ class Controller(object):
 
                 # Evaluate objective
                 x = self.model.as_absolute_coordinates(new_point)
-                rvec_list, f_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
+                rvec_list, obj_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
 
                 # Handle exit conditions (f < min obj value or maxfun reached)
                 if exit_info is not None:
@@ -398,7 +403,7 @@ class Controller(object):
         for j in range(num_steps):
             xnew = self.model.xopt() + (step_length / LA.norm(dirns[j, :])) * dirns[j, :]
             x = self.model.as_absolute_coordinates(xnew)
-            rvec_list, f_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
+            rvec_list, obj_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
 
             # Handle exit conditions (f < min obj value or maxfun reached)
             if exit_info is not None:
@@ -436,13 +441,85 @@ class Controller(object):
 
         return dirn * (step_length / LA.norm(dirn))
 
-    def trust_region_step(self, params):
-        # Build model for full least squares objectives
+    def evaluate_criticality_measure(self, params):
+        # Calculate criticality measure for regularized problems (h is not None)
+        
+        # Build model for full least squares function
         gopt, H = self.model.build_full_model()
+        
+        if np.any(np.isnan(gopt)) or np.any(np.isnan(H)) or not np.all(np.isfinite(gopt)) or not np.all(np.isfinite(H)):
+            module_logger.debug("nan/inf values in gopt and/or H, skipping ctrsbox_sfista (criticality measure calc)")
+            # d = np.zeros(gopt.shape)
+            # gnew = gopt.copy()
+            # crvmin = -1
+            return np.inf
+        
+        # NOTE: smaller params here to get more iterations in S-FISTA
+        func_tol = params("func_tol.criticality_measure") * self.delta
         if self.model.projections:
-            d, gnew, crvmin = ctrsbox(self.model.xopt(abs_coordinates=True), gopt, H, self.model.projections, self.delta, d_max_iters=params("dykstra.max_iters"), d_tol=params("dykstra.d_tol"))
+            d, gnew, crvmin = ctrsbox_sfista(self.model.xopt(abs_coordinates=True), gopt, np.zeros(H.shape), self.model.projections, 1,
+                                self.h, self.lh, self.prox_uh, argsh = self.argsh, argsprox=self.argsprox, func_tol=func_tol, 
+                                max_iters=params("func_tol.max_iters"), d_max_iters=params("dykstra.max_iters"), d_tol=params("dykstra.d_tol"),
+                                scaling_changes=self.scaling_changes, sfista_iters_scale=params("sfista.max_iters_scaling"))
+        else:
+            proj = lambda x: pbox(x, self.model.sl, self.model.su)
+            d, gnew, crvmin = ctrsbox_sfista(self.model.xopt(abs_coordinates=True), gopt, np.zeros(H.shape), [proj], 1,
+                                self.h, self.lh, self.prox_uh, argsh = self.argsh, argsprox=self.argsprox, func_tol=func_tol, 
+                                max_iters=params("func_tol.max_iters"), d_max_iters=params("dykstra.max_iters"), d_tol=params("dykstra.d_tol"),
+                                scaling_changes=self.scaling_changes, sfista_iters_scale=params("sfista.max_iters_scaling"))
+        
+        # Calculate criticality measure
+        criticality_measure = self.h(remove_scaling(self.model.xopt(abs_coordinates=True), self.scaling_changes), *self.argsh) - model_value(gopt, np.zeros(H.shape), d, self.model.xopt(abs_coordinates=True), self.h, self.argsh, self.scaling_changes)
+        return criticality_measure
+
+    def trust_region_step(self, params, criticality_measure=1e-2):
+        # Build model for full least squares function
+        gopt, H = self.model.build_full_model()
+        # Build func_tol for trust region step
+        # QUESTION: c1 = min{1, 1/delta_max^2}, but choose c1=1here; choose maxhessian = max(||H||_2,1)
+        # QUESTION: when criticality_measure = 0? choose max(criticality_measure,1)
+        func_tol = (1-params("func_tol.tr_step")) * 1 * max(criticality_measure,1) * min(self.delta, max(criticality_measure,1) / max(np.linalg.norm(H, 2),1))
+
+        if self.h is None:
+            if self.model.projections:
+                # Running PGD/SFISTA is generally slower than trsbox, so don't do this if gopt or H have bad values
+                # (this will ultimately lead to a manual setting of d=0 and calling a safety step anyway)
+                if np.any(np.isnan(gopt)) or np.any(np.isnan(H)) or not np.all(np.isfinite(gopt)) or not np.all(np.isfinite(H)):
+                    module_logger.debug("nan/inf values in gopt and/or H, skipping ctrsbox_pgd")
+                    d = np.zeros(gopt.shape)
+                    gnew = gopt.copy()
+                    crvmin = -1
+                else:
+                    d, gnew, crvmin = ctrsbox_pgd(self.model.xopt(abs_coordinates=True), gopt, H, self.model.projections, self.delta, d_max_iters=params("dykstra.max_iters"), d_tol=params("dykstra.d_tol"))
+            else:
+                d, gnew, crvmin = trsbox(self.model.xopt(), gopt, H, self.model.sl, self.model.su, self.delta)
         else:
-            d, gnew, crvmin = trsbox(self.model.xopt(), gopt, H, self.model.sl, self.model.su, self.delta)
+            # Running PGD/SFISTA is generally slower than trsbox, so don't do this if gopt or H have bad values
+            # (this will ultimately lead to a manual setting of d=0 and calling a safety step anyway)
+            if np.any(np.isnan(gopt)) or np.any(np.isnan(H)) or not np.all(np.isfinite(gopt)) or not np.all(np.isfinite(H)):
+                module_logger.debug("nan/inf values in gopt and/or H, skipping ctrsbox_sfista")
+                d = np.zeros(gopt.shape)
+                gnew = gopt.copy()
+                crvmin = -1
+            elif self.model.projections:
+                d, gnew, crvmin = ctrsbox_sfista(self.model.xopt(abs_coordinates=True), gopt, H, self.model.projections, self.delta,
+                                     self.h, self.lh, self.prox_uh, argsh = self.argsh, argsprox=self.argsprox, func_tol=func_tol, 
+                                     max_iters=params("func_tol.max_iters"), d_max_iters=params("dykstra.max_iters"), d_tol=params("dykstra.d_tol"),
+                                     scaling_changes=self.scaling_changes, sfista_iters_scale=params("sfista.max_iters_scaling"))
+            else:
+                # NOTE: alternative way if using trsbox
+                # d, gnew, crvmin = trsbox(self.model.xopt(), gopt, H, self.model.sl, self.model.su, self.delta)
+                proj = lambda x: pbox(x, self.model.sl, self.model.su)
+                d, gnew, crvmin = ctrsbox_sfista(self.model.xopt(abs_coordinates=True), gopt, H, [proj], self.delta,
+                                      self.h, self.lh, self.prox_uh, argsh = self.argsh, argsprox=self.argsprox, func_tol=func_tol,
+                                      max_iters=params("func_tol.max_iters"), d_max_iters=params("dykstra.max_iters"), d_tol=params("dykstra.d_tol"),
+                                      scaling_changes=self.scaling_changes, sfista_iters_scale=params("sfista.max_iters_scaling"))
+            
+            # NOTE: check sufficient decrease. If increase in the model, set zero step
+            pred_reduction = self.h(remove_scaling(self.model.xopt(abs_coordinates=True), self.scaling_changes), *self.argsh) - model_value(gopt, H, d, self.model.xopt(abs_coordinates=True), self.h, self.argsh, self.scaling_changes)
+            if pred_reduction < 0.0:
+                d = np.zeros(d.shape)
+        
         return d, gopt, H, gnew, crvmin
 
     def geometry_step(self, knew, adelt, number_of_samples, params):
@@ -463,10 +540,10 @@ class Controller(object):
             return exit_info  # didn't fix geometry - return & quit
 
         gopt, H = self.model.build_full_model()  # save here, to calculate predicted value from geometry step
-        fopt = self.model.fopt()  # again, evaluate now, before model.change_point()
+        objopt = self.model.objopt()  # again, evaluate now, before model.change_point()
         d = xnew - self.model.xopt()
         x = self.model.as_absolute_coordinates(xnew)
-        rvec_list, f_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
+        rvec_list, obj_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
 
         # Handle exit conditions (f < min obj value or maxfun reached)
         if exit_info is not None:
@@ -481,11 +558,14 @@ class Controller(object):
             self.model.add_new_sample(knew, rvec_extra=rvec_list[i, :])
 
         # Estimate actual reduction to add to diffs vector
-        f = sumsq(np.mean(rvec_list[:num_samples_run, :], axis=0))  # estimate actual objective value
-
+        obj = sumsq(np.mean(rvec_list[:num_samples_run, :], axis=0)) # estimate actual objective value
         # pred_reduction = - calculate_model_value(gopt, H, d)
         pred_reduction = - model_value(gopt, H, d)
-        actual_reduction = fopt - f
+        if self.h is not None:
+            obj += self.h(remove_scaling(x, self.scaling_changes), *self.argsh)
+            # since m(0) = h(x)
+            pred_reduction = self.h(remove_scaling(x, self.scaling_changes), *self.argsh) - model_value(gopt, H, d, x, self.h, self.argsh, self.scaling_changes)
+        actual_reduction = objopt - obj
         self.diffs = [abs(pred_reduction - actual_reduction), self.diffs[0], self.diffs[1]]
         return None  # exit_info = None
 
@@ -513,7 +593,7 @@ class Controller(object):
     def evaluate_objective(self, x, number_of_samples, params):
         # Sample from objective function several times, keeping track of maxfun and min_obj_value throughout
         rvec_list = np.zeros((number_of_samples, self.m()))
-        f_list = np.zeros((number_of_samples,))
+        obj_list = np.zeros((number_of_samples,))
         num_samples_run = 0
         incremented_nx = False
         exit_info = None
@@ -527,19 +607,24 @@ class Controller(object):
             if not incremented_nx:
                 self.nx += 1
                 incremented_nx = True
-            rvec_list[i, :], f_list[i] = eval_least_squares_objective(self.objfun, remove_scaling(x, self.scaling_changes),
-                                            args=self.args, eval_num=self.nf, pt_num=self.nx,
+            rvec_list[i, :], obj_list[i] = eval_least_squares_with_regularisation(self.objfun, remove_scaling(x, self.scaling_changes), self.h,
+                                            argsf=self.argsf, argsh=self.argsh, verbose=self.do_logging, eval_num=self.nf, pt_num=self.nx,
                                             full_x_thresh=params("logging.n_to_print_whole_x_vector"),
-                                            check_for_overflow=params("general.check_objfun_for_overflow"),
-                                            verbose=self.do_logging)
+                                            check_for_overflow=params("general.check_objfun_for_overflow"))
             num_samples_run += 1
 
         # Check if the average value was below our threshold
-        if num_samples_run > 0 and \
-                        sumsq(np.mean(rvec_list[:num_samples_run, :], axis=0)) <= self.model.min_objective_value():
-            exit_info = ExitInformation(EXIT_SUCCESS, "Objective is sufficiently small")
+        # QUESTION: how to choose x in h when using averaged values
+        if self.h is None:
+            if num_samples_run > 0 and \
+                            sumsq(np.mean(rvec_list[:num_samples_run, :], axis=0)) <= self.model.min_objective_value():
+                exit_info = ExitInformation(EXIT_SUCCESS, "Objective is sufficiently small")
+        else:
+            if num_samples_run > 0 and \
+                            sumsq(np.mean(rvec_list[:num_samples_run, :], axis=0)) + self.h(remove_scaling(x, self.scaling_changes),*self.argsh) <= self.model.min_objective_value():
+                exit_info = ExitInformation(EXIT_SUCCESS, "Objective is sufficiently small")
 
-        return rvec_list, f_list, num_samples_run, exit_info
+        return rvec_list, obj_list, num_samples_run, exit_info
 
     def choose_point_to_replace(self, d, skip_kopt=True):
         delsq = self.delta ** 2
@@ -615,11 +700,18 @@ class Controller(object):
         self.last_successful_iter = current_iter  # reset successful iteration check
         return
 
-    def calculate_ratio(self, current_iter, rvec_list, d, gopt, H):
+    def calculate_ratio(self, x, current_iter, rvec_list, d, gopt, H):
         exit_info = None
-        f = sumsq(np.mean(rvec_list, axis=0))  # estimate actual objective value
-        pred_reduction = - model_value(gopt, H, d) # negative of m since m(0) = 0
-        actual_reduction = self.model.fopt() - f
+        # estimate actual objective value
+        obj = sumsq(np.mean(rvec_list, axis=0))
+        # pred_reduction = - calculate_model_value(gopt, H, d)
+        pred_reduction = - model_value(gopt, H, d)
+        if self.h is not None:
+            # QUESTION: x+d here correct? rvec_list takes mean value
+            obj += self.h(remove_scaling(x+d, self.scaling_changes), *self.argsh) 
+            # since m(0) = h(x)
+            pred_reduction = self.h(remove_scaling(x, self.scaling_changes), *self.argsh) - model_value(gopt, H, d, x, self.h, self.argsh, self.scaling_changes)
+        actual_reduction = self.model.objopt() - obj
         self.diffs = [abs(actual_reduction - pred_reduction), self.diffs[0], self.diffs[1]]
         if min(sqrt(sumsq(d)), self.delta) > self.rho:  # if ||d|| >= rho, successful!
             self.last_successful_iter = current_iter
@@ -627,8 +719,7 @@ class Controller(object):
             if len(self.model.projections) > 1: # if we are using multiple projections, only warn since likely due to constraint intersection
                 exit_info = ExitInformation(EXIT_TR_INCREASE_WARNING, "Either multiple constraints are active or trust region step gave model increase")
             else:
-                exit_info = ExitInformation(EXIT_TR_INCREASE_ERROR, "Either rust region step gave model increase")
-
+                exit_info = ExitInformation(EXIT_TR_INCREASE_ERROR, "Trust region step gave model increase")
         ratio = actual_reduction / pred_reduction
         return ratio, exit_info
 
@@ -636,13 +727,13 @@ class Controller(object):
         if len(self.last_iters_step_taken) <= params("slow.history_for_slow"):
             # Not enough info, simply append
             self.last_iters_step_taken.append(current_iter)
-            self.last_fopts_step_taken.append(self.model.fopt())
+            self.last_fopts_step_taken.append(self.model.objopt())
             this_iter_slow = False
         else:
             # Enough info - shift values
             self.last_iters_step_taken = self.last_iters_step_taken[1:] + [current_iter]
-            self.last_fopts_step_taken = self.last_fopts_step_taken[1:] + [self.model.fopt()]
-            this_iter_slow = (log(self.last_fopts_step_taken[0]) - log(self.model.fopt())) / \
+            self.last_fopts_step_taken = self.last_fopts_step_taken[1:] + [self.model.objopt()]
+            this_iter_slow = (log(self.last_fopts_step_taken[0]) - log(self.model.objopt())) / \
                              float(params("slow.history_for_slow")) < params("slow.thresh_for_slow")
         # Update counter of number of slow iterations
         if this_iter_slow:
@@ -659,9 +750,9 @@ class Controller(object):
     def soft_restart(self, number_of_samples, nruns_so_far, params, x_in_abs_coords_to_save=None, rvec_to_save=None,
                      nsamples_to_save=None):
         # A successful run is one where we reduced fopt
-        if self.model.fopt() < self.last_run_fopt:
+        if self.model.objopt() < self.last_run_fopt:
             self.last_successful_run = nruns_so_far
-        self.last_run_fopt = self.model.fopt()
+        self.last_run_fopt = self.model.objopt()
 
         ok_to_do_restart = (nruns_so_far - self.last_successful_run < params("restarts.max_unsuccessful_restarts")) and \
                            (self.nf < self.maxfun)
@@ -682,7 +773,7 @@ class Controller(object):
                               self.model.nsamples[self.model.kopt], x_in_abs_coords=True)
 
         if self.do_logging:
-            module_logger.info("Soft restart [currently, f = %g after %g function evals]" % (self.model.fopt(), self.nf))
+            module_logger.info("Soft restart [currently, f = %g after %g function evals]" % (self.model.objopt(), self.nf))
         # Resetting method: reset delta and rho, then move the closest 'num_steps' points to xk to improve geometry
         # Note: closest points because we are suddenly increasing delta & rho, so we want to encourage spreading out points
         self.delta = self.rhobeg
@@ -724,7 +815,7 @@ class Controller(object):
             for i in range(num_pts_to_add):
                 xnew = self.model.xopt() + dirns[i, :]  # always base move around best value so far
                 x = self.model.as_absolute_coordinates(xnew)
-                rvec_list, f_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
+                rvec_list, obj_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
 
                 # Handle exit conditions (f < min obj value or maxfun reached)
                 if exit_info is not None:
@@ -771,11 +862,11 @@ class Controller(object):
             add_noise = params("noise.scale_factor_for_quit") * params("noise.additive_noise_level")
             for k in range(self.model.npt()):
                 all_fvals_within_noise = all_fvals_within_noise and \
-                                (self.model.fval[k] <= self.model.fopt() + add_noise / sqrt(self.model.nsamples[k]))
+                                (self.model.objval[k] <= self.model.objopt() + add_noise / sqrt(self.model.nsamples[k]))
         else:  # noise_level_multiplicative
             ratio = 1.0 + params("noise.scale_factor_for_quit") * params("noise.multiplicative_noise_level")
             for k in range(self.model.npt()):
-                this_ratio = self.model.fval[k] / self.model.fopt()  # fval_opt strictly positive (would have quit o/w)
+                this_ratio = self.model.objval[k] / self.model.objopt()  # fval_opt strictly positive (would have quit o/w)
                 all_fvals_within_noise = all_fvals_within_noise and (
                     this_ratio <= ratio / sqrt(self.model.nsamples[k]))
         return all_fvals_within_noise
@@ -804,7 +895,7 @@ class Controller(object):
                     dirns[i, :] = -dirns[i, :]
             xnew = np.maximum(np.minimum(self.model.xopt() + dirns[i, :], self.model.su), self.model.sl)
             x = self.model.as_absolute_coordinates(xnew)
-            rvec_list, f_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
+            rvec_list, obj_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
 
             # Handle exit conditions (f < min obj value or maxfun reached)
             if exit_info is not None:

dfols/model.py

@@ -36,7 +36,7 @@ import numpy as np
 import scipy.linalg as LA
 
 from .trust_region import trsbox_geometry
-from .util import sumsq, dykstra
+from .util import sumsq, dykstra, remove_scaling
 
 __all__ = ['Model']
 
@@ -44,8 +44,8 @@ module_logger = logging.getLogger(__name__)
 
 
 class Model(object):
-    def __init__(self, npt, x0, r0, xl, xu, projections, r0_nsamples, n=None, m=None, abs_tol=1e-12, rel_tol=1e-20, precondition=True,
-                 do_logging=True):
+    def __init__(self, npt, x0, r0, xl, xu, projections, r0_nsamples, h=None, argsh=(), n=None, m=None, abs_tol=1e-12, rel_tol=1e-20, precondition=True,
+                 do_logging=True, scaling_changes=None):
         if n is None:
             n = len(x0)
         if m is None:
@@ -56,11 +56,15 @@ class Model(object):
         assert xu.shape == (n,), "xu has wrong shape (got %s, expect (%g,))" % (str(xu.shape), n)
         assert r0.shape == (m,), "r0 has wrong shape (got %s, expect (%g,))" % (str(r0.shape), m)
         self.do_logging = do_logging
+        self.scaling_changes = scaling_changes
         self.dim = n
         self.resid_dim = m
         self.num_pts = npt
         self.npt_so_far = 1  # number of points added so far (with function values)
 
+        self.h = h
+        self.argsh = argsh
+
         # Initialise to blank some useful stuff
         # Interpolation points
         self.xbase = x0.copy()
@@ -72,12 +76,15 @@ class Model(object):
         # Function values
         self.fval_v = np.inf * np.ones((npt, m))  # residuals for each xpt
         self.fval_v[0, :] = r0.copy()
-        self.fval = np.inf * np.ones((npt, ))  # overall objective value for each xpt
-        self.fval[0] = sumsq(r0)
+        
+        self.objval = np.inf * np.ones((npt, ))  # overall objective value for each xpt
+        self.objval[0] = sumsq(r0)
+        if h is not None:
+            self.objval[0] += h(remove_scaling(x0, self.scaling_changes), *argsh)
         self.kopt = 0  # index of current iterate (should be best value so far)
         self.nsamples = np.zeros((npt,), dtype=int)  # number of samples used to evaluate objective at each point
         self.nsamples[0] = r0_nsamples
-        self.fbeg = self.fval[0]  # f(x0), saved to check for sufficient reduction
+        self.objbeg = self.objval[0]  # f(x0), saved to check for sufficient reduction
 
         # Termination criteria
         self.abs_tol = abs_tol
@@ -90,7 +97,7 @@ class Model(object):
         # Saved point (in absolute coordinates) - always check this value before quitting solver
         self.xsave = None
         self.rsave = None
-        self.fsave = None
+        self.objsave = None
         self.jacsave = None
         self.nsamples_save = None
 
@@ -118,8 +125,8 @@ class Model(object):
     def ropt(self):
         return self.fval_v[self.kopt, :]  # residuals for current iterate
 
-    def fopt(self):
-        return self.fval[self.kopt]
+    def objopt(self):
+        return self.objval[self.kopt]
 
     def xpt(self, k, abs_coordinates=False):
         assert 0 <= k < self.npt(), "Invalid index %g" % k
@@ -135,9 +142,9 @@ class Model(object):
         assert 0 <= k < self.npt(), "Invalid index %g" % k
         return self.fval_v[k, :]
 
-    def fval(self, k):
+    def objval(self, k):
         assert 0 <= k < self.npt(), "Invalid index %g" % k
-        return self.fval[k]
+        return self.objval[k]
 
     def as_absolute_coordinates(self, x, full_dykstra=False):
         # If x were an interpolation point, get the absolute coordinates of x
@@ -177,18 +184,20 @@ class Model(object):
 
         self.points[k, :] = x.copy()
         self.fval_v[k, :] = rvec.copy()
-        self.fval[k] = sumsq(rvec)
+        self.objval[k] = sumsq(rvec)
+        if self.h is not None:
+            self.objval[k] += self.h(remove_scaling(self.xbase + x, self.scaling_changes), *self.argsh)
         self.nsamples[k] = 1
         self.factorisation_current = False
 
-        if allow_kopt_update and self.fval[k] < self.fopt():
+        if allow_kopt_update and self.objval[k] < self.objopt():
             self.kopt = k
         return
 
     def swap_points(self, k1, k2):
         self.points[[k1, k2], :] = self.points[[k2, k1], :]
         self.fval_v[[k1, k2], :] = self.fval_v[[k2, k1], :]
-        self.fval[[k1, k2]] = self.fval[[k2, k1]]
+        self.objval[[k1, k2]] = self.objval[[k2, k1]]
         if self.kopt == k1:
             self.kopt = k2
         elif self.kopt == k2:
@@ -201,22 +210,27 @@ class Model(object):
         assert 0 <= k < self.npt(), "Invalid index %g" % k
         t = float(self.nsamples[k]) / float(self.nsamples[k] + 1)
         self.fval_v[k, :] = t * self.fval_v[k, :] + (1 - t) * rvec_extra
-        self.fval[k] = sumsq(self.fval_v[k, :])
+        # NOTE: how to sample when we have h? still at xpt(k), then add h(xpt(k)). Modify test if incorrect!
+        self.objval[k] = sumsq(self.fval_v[k, :])
+        if self.h is not None:
+            self.objval[k] += self.h(remove_scaling(self.xbase + self.points[k, :], self.scaling_changes), *self.argsh)
         self.nsamples[k] += 1
 
-        self.kopt = np.argmin(self.fval[:self.npt()])  # make sure kopt is always the best value we have
+        self.kopt = np.argmin(self.objval[:self.npt()])  # make sure kopt is always the best value we have
         return
 
     def add_new_point(self, x, rvec):
         self.points = np.append(self.points, x.reshape((1, self.n())), axis=0)  # append row to xpt
         self.fval_v = np.append(self.fval_v, rvec.reshape((1, self.m())), axis=0)  # append row to fval_v
-        f = np.dot(rvec, rvec)
-        self.fval = np.append(self.fval, f)  # append entry to fval
+        obj = sumsq(rvec)
+        if self.h is not None:
+            obj += self.h(remove_scaling(self.xbase + x, self.scaling_changes), *self.argsh)
+        self.objval = np.append(self.objval, obj)  # append entry to fval
         self.nsamples = np.append(self.nsamples, 1)  # add new sample number
         self.num_pts += 1  # make sure npt is updated
         self.npt_so_far += 1
 
-        if f < self.fopt():
+        if obj < self.objopt():
             self.kopt = self.npt() - 1
 
         self.factorisation_current = False
@@ -236,11 +250,14 @@ class Model(object):
         return
 
     def save_point(self, x, rvec, nsamples, x_in_abs_coords=True):
-        f = sumsq(rvec)
-        if self.fsave is None or f <= self.fsave:
-            self.xsave = x.copy() if x_in_abs_coords else self.as_absolute_coordinates(x)
+        xabs = x.copy() if x_in_abs_coords else self.as_absolute_coordinates(x)
+        obj = sumsq(rvec)
+        if self.h is not None:
+            obj += self.h(remove_scaling(xabs, self.scaling_changes), *self.argsh)
+        if self.objsave is None or obj <= self.objsave:
+            self.xsave = xabs
             self.rsave = rvec.copy()
-            self.fsave = f
+            self.objsave = obj
             self.jacsave = self.model_jac.copy()
             self.nsamples_save = nsamples
             return True
@@ -248,15 +265,15 @@ class Model(object):
             return False  # this value is worse than what we have already - didn't save
 
     def get_final_results(self):
-        # Return x and fval for optimal point (either from xsave+fsave or kopt)
-        if self.fsave is None or self.fopt() <= self.fsave:  # optimal has changed since xsave+fsave were last set
-            return self.xopt(abs_coordinates=True).copy(), self.ropt().copy(), self.fopt(), self.model_jac.copy(), self.nsamples[self.kopt]
+        # Return x and objval for optimal point (either from xsave+objsave or kopt)
+        if self.objsave is None or self.objopt() <= self.objsave:  # optimal has changed since xsave+objsave were last set
+            return self.xopt(abs_coordinates=True).copy(), self.ropt().copy(), self.objopt(), self.model_jac.copy(), self.nsamples[self.kopt]
         else:
-            return self.xsave.copy(), self.rsave.copy(), self.fsave, self.jacsave, self.nsamples_save
+            return self.xsave.copy(), self.rsave.copy(), self.objsave, self.jacsave, self.nsamples_save
 
     def min_objective_value(self):
         # Get termination criterion for f small: f <= abs_tol or f <= rel_tol * f0
-        return max(self.abs_tol, self.rel_tol * self.fbeg)
+        return max(self.abs_tol, self.rel_tol * self.objbeg)
 
     def model_value(self, d, d_based_at_xopt=True, with_const_term=False):
         if d_based_at_xopt:
@@ -375,7 +392,7 @@ class Model(object):
         return True, interp_error, sqrt(norm_J_error), linalg_resid, ls_interp_cond_num  # flag ok
 
     def build_full_model(self):
-        # Build full least squares objective model from mini-models
+        # Build full least squares model from mini-models
         # Centred around xopt
         r = self.model_const + np.dot(self.model_jac, self.xopt())  # constant term (for inexact interpolation)
         J = self.model_jac

dfols/params.py

@@ -82,7 +82,7 @@ class ParameterList(object):
         self.params["restarts.use_soft_restarts"] = True
         self.params["restarts.soft.num_geom_steps"] = 3
         self.params["restarts.soft.move_xk"] = True
-        self.params["restarts.soft.max_fake_successful_steps"] = maxfun  # number ratio>0 steps below fsave allowed
+        self.params["restarts.soft.max_fake_successful_steps"] = maxfun  # number ratio>0 steps below objsave allowed
         self.params["restarts.hard.use_old_rk"] = True  # recycle r(xk) from previous run?
         self.params["restarts.increase_npt"] = False
         self.params["restarts.increase_npt_amt"] = 1
@@ -109,12 +109,20 @@ class ParameterList(object):
         self.params["growing.full_rank.min_sing_val"] = 1e-6  # absolute floor on singular values
         self.params["growing.full_rank.svd_max_jac_cond"] = 1e8  # maximum condition number of Jacobian
         self.params["growing.perturb_trust_region_step"] = False  # add random direction onto TRS solution?
+        
         # Dykstra's algorithm
         self.params["dykstra.d_tol"] = 1e-10
         self.params["dykstra.max_iters"] = 100
+        
         # Matrix rank algorithm
         self.params["matrix_rank.r_tol"] = 1e-18
-
+        
+        # Function tolerance when applying S-FISTA method
+        self.params["func_tol.criticality_measure"] = 1e-3
+        self.params["func_tol.tr_step"] = 1-1e-1
+        self.params["func_tol.max_iters"] = 500
+        self.params["sfista.max_iters_scaling"] = 2.0
+        
         self.params_changed = {}
         for p in self.params:
             self.params_changed[p] = False
@@ -268,6 +276,14 @@ class ParameterList(object):
             type_str, nonetype_ok, lower, upper = 'int', False, 0, None
         elif key == "matrix_rank.r_tol":
             type_str, nonetype_ok, lower, upper = 'float', False, 0.0, None
+        elif key == "func_tol.criticality_measure":
+            type_str, nonetype_ok, lower, upper = 'float', False, 0.0, 1.0
+        elif key == "func_tol.tr_step":
+            type_str, nonetype_ok, lower, upper = 'float', False, 0.0, 1.0
+        elif key == "func_tol.max_iters":
+            type_str, nonetype_ok, lower, upper = 'int', False, 0, None
+        elif key == "sfista.max_iters_scaling":
+            type_str, nonetype_ok, lower, upper = 'float', False, 1.0, None
         else:
             assert False, "ParameterList.param_type() has unknown key: %s" % key
         return type_str, nonetype_ok, lower, upper

dfols/solver.py

@@ -48,10 +48,10 @@ module_logger = logging.getLogger(__name__)
 
 # A container for the results of the optimization routine
 class OptimResults(object):
-    def __init__(self, xmin, rmin, fmin, jacmin, nf, nx, nruns, exit_flag, exit_msg):
+    def __init__(self, xmin, rmin, objmin, jacmin, nf, nx, nruns, exit_flag, exit_msg):
         self.x = xmin
         self.resid = rmin
-        self.f = fmin
+        self.obj = objmin
         self.jacobian = jacmin
         self.nf = nf
         self.nx = nx
@@ -77,7 +77,7 @@ class OptimResults(object):
                 output += "Residual vector = %s\n" % str(self.resid)
             else:
                 output += "Not showing residual vector because it is too long; check self.resid\n"
-            output += "Objective value f(xmin) = %.10g\n" % self.f
+            output += "Objective value f(xmin) = %.10g\n" % self.obj
             output += "Needed %g objective evaluations (at %g points)\n" % (self.nf, self.nx)
             if self.nruns > 1:
                 output += "Did a total of %g runs\n" % self.nruns
@@ -95,8 +95,8 @@ class OptimResults(object):
         return output
 
 
-def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfun, nruns_so_far, nf_so_far, nx_so_far, nsamples, params,
-               diagnostic_info, scaling_changes, r0_avg_old=None, r0_nsamples_old=None, default_growing_method_set_by_user=None,
+def solve_main(objfun, x0, argsf, xl, xu, projections, npt, rhobeg, rhoend, maxfun, nruns_so_far, nf_so_far, nx_so_far, nsamples, params,
+               diagnostic_info, scaling_changes, h=None, lh=None, argsh=(), prox_uh=None, argsprox=None, r0_avg_old=None, r0_nsamples_old=None, default_growing_method_set_by_user=None,
                do_logging=True, print_progress=False):
     # Evaluate at x0 (keep nf, nx correct and check for f < 1e-12)
     # The hard bit is determining what m = len(r0) should be, and allocating memory appropriately
@@ -105,18 +105,17 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
         # Evaluate the first time...
         nf = nf_so_far + 1
         nx = nx_so_far + 1
-        r0, f0 = eval_least_squares_objective(objfun, remove_scaling(x0, scaling_changes),
-                                              args=args, eval_num=nf, pt_num=nx,
+        r0, obj0 = eval_least_squares_with_regularisation(objfun, remove_scaling(x0, scaling_changes), h, 
+                                              argsf=argsf, argsh=argsh, verbose=do_logging, eval_num=nf, pt_num=nx,
                                               full_x_thresh=params("logging.n_to_print_whole_x_vector"),
-                                              check_for_overflow=params("general.check_objfun_for_overflow"),
-                                              verbose=do_logging)
+                                              check_for_overflow=params("general.check_objfun_for_overflow"))
         m = len(r0)
 
         # Now we have m, we can evaluate the rest of the times
         rvec_list = np.zeros((number_of_samples, m))
-        f_list = np.zeros((number_of_samples,))
+        obj_list = np.zeros((number_of_samples,))
         rvec_list[0, :] = r0
-        f_list[0] = f0
+        obj_list[0] = obj0
         num_samples_run = 1
         exit_info = None
 
@@ -128,15 +127,20 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
 
             nf += 1
             # Don't increment nx for x0 - we did this earlier
-            rvec_list[i, :], f_list[i] = eval_least_squares_objective(objfun, remove_scaling(x0, scaling_changes), args=args, eval_num=nf, pt_num=nx,
+            rvec_list[i, :], obj_list[i] = eval_least_squares_with_regularisation(objfun, remove_scaling(x0, scaling_changes), h, 
+                                                argsf=argsf, argsh=argsh, verbose=do_logging, eval_num=nf, pt_num=nx,
                                                 full_x_thresh=params("logging.n_to_print_whole_x_vector"),
-                                                check_for_overflow=params("general.check_objfun_for_overflow"),
-                                                verbose=do_logging)
+                                                check_for_overflow=params("general.check_objfun_for_overflow"))
             num_samples_run += 1
 
         r0_avg = np.mean(rvec_list[:num_samples_run, :], axis=0)
-        if sumsq(r0_avg) <= params("model.abs_tol"):
-            exit_info = ExitInformation(EXIT_SUCCESS, "Objective is sufficiently small")
+        # NOTE: modify objvalue here
+        if h is None:
+            if sumsq(r0_avg) <= params("model.abs_tol"):
+                exit_info = ExitInformation(EXIT_SUCCESS, "Objective is sufficiently small")
+        else:
+            if sumsq(r0_avg) + h(remove_scaling(x0, scaling_changes), *argsh)<= params("model.abs_tol"):
+                exit_info = ExitInformation(EXIT_SUCCESS, "Objective is sufficiently small")
 
         if exit_info is not None:
             return x0, r0_avg, sumsq(r0_avg), None, num_samples_run, nf, nx, nruns_so_far+1, exit_info, diagnostic_info
@@ -162,8 +166,8 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
                 params('growing.delta_scale_new_dirns', new_value=0.1)
 
     # Initialise controller
-    control = Controller(objfun, args, x0, r0_avg, num_samples_run, xl, xu, projections, npt, rhobeg, rhoend, nf, nx, maxfun,
-                         params, scaling_changes, do_logging)
+    control = Controller(objfun, argsf, x0, r0_avg, num_samples_run, xl, xu, projections, npt, rhobeg, rhoend, nf, nx, maxfun,
+                         params, scaling_changes, do_logging, h=h, lh=lh, argsh=argsh,  prox_uh=prox_uh, argsprox=argsprox)
 
     # Initialise interpolation set
     number_of_samples = max(nsamples(control.delta, control.rho, 0, nruns_so_far), 1)
@@ -178,8 +182,8 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
             module_logger.info("Initialising (coordinate directions)")
         exit_info = control.initialise_coordinate_directions(number_of_samples, num_directions, params)
     if exit_info is not None:
-        x, rvec, f, jacmin, nsamples = control.model.get_final_results()
-        return x, rvec, f, None, nsamples, control.nf, control.nx, nruns_so_far + 1, exit_info, diagnostic_info
+        x, rvec, obj, jacmin, nsamples = control.model.get_final_results()
+        return x, rvec, obj, None, nsamples, control.nf, control.nx, nruns_so_far + 1, exit_info, diagnostic_info
 
     finished_growing = (control.model.npt() >= control.model.num_pts)  # have we finished growing the initial set yet?
 
@@ -271,16 +275,30 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
                 nruns_so_far += 1
                 break  # quit
 
-
-        # Trust region step
-        d, gopt, H, gnew, crvmin = control.trust_region_step(params)
+        tau = 1.0 # ratio used in the safety phase
+        if h is None:
+            # Trust region step
+            d, gopt, H, gnew, crvmin = control.trust_region_step(params)
+        else:
+            # Calculate criticality measure
+            criticality_measure = control.evaluate_criticality_measure(params)
+            # Trust region step
+            d, gopt, H, gnew, crvmin = control.trust_region_step(params, criticality_measure)
+            try:
+                tau = min(criticality_measure/(LA.norm(gopt)+lh), 1.0)
+            except ValueError:
+                # In some instances, gopt can have nan/inf values -- this ultimately calls a safety step and is generally fine
+                # but we need to set a value for tau nonetheless
+                tau = 1.0
+        
         if do_logging:
             module_logger.debug("Trust region step is d = " + str(d))
+        
         xnew = control.model.xopt() + d
         dnorm = min(LA.norm(d), control.delta)
 
         if print_progress:
-            print("{:^5}{:^7}{:^10.2e}{:^10.2e}{:^10.2e}{:^10.2e}{:^7}".format(nruns_so_far+1, current_iter+1, control.model.fopt(), np.linalg.norm(gopt), control.delta, control.rho, control.nf))
+            print("{:^5}{:^7}{:^10.2e}{:^10.2e}{:^10.2e}{:^10.2e}{:^7}".format(nruns_so_far+1, current_iter+1, control.model.objopt(), np.linalg.norm(gopt), control.delta, control.rho, control.nf))
 
         if params("logging.save_diagnostic_info"):
             diagnostic_info.save_info_from_control(control, nruns_so_far, current_iter,
@@ -289,7 +307,7 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
             diagnostic_info.update_interpolation_information(interp_error, ls_interp_cond_num, linalg_resid,
                                                              sqrt(norm_J_error), LA.norm(gopt), LA.norm(d))
 
-        if dnorm < params("general.safety_step_thresh") * control.rho and not finished_growing and params("growing.safety.do_safety_step"):
+        if dnorm < tau * params("general.safety_step_thresh") * control.rho and not finished_growing and params("growing.safety.do_safety_step"):
             if do_logging:
                 module_logger.debug("Safety step during growing phase")
 
@@ -415,10 +433,10 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
                 if do_logging:
                     module_logger.info("New rho = %g after %i function evaluations" % (control.rho, control.nf))
                     if control.n() < params("logging.n_to_print_whole_x_vector"):
-                        module_logger.debug("Best so far: f = %.15g at x = " % (control.model.fopt())
+                        module_logger.debug("Best so far: f = %.15g at x = " % (control.model.objopt())
                                       + str(control.model.xopt(abs_coordinates=True)))
                     else:
-                        module_logger.debug("Best so far: f = %.15g at x = [...]" % (control.model.fopt()))
+                        module_logger.debug("Best so far: f = %.15g at x = [...]" % (control.model.objopt()))
                 continue  # next iteration
             else:
                 # Quit on rho=rhoend
@@ -439,8 +457,9 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
                 else:
                     # Cannot reduce rho, so check xnew and quit
                     x = control.model.as_absolute_coordinates(xnew)
+                    ##print("x from xnew", x)
                     number_of_samples = max(nsamples(control.delta, control.rho, current_iter, nruns_so_far), 1)
-                    rvec_list, f_list, num_samples_run, exit_info = control.evaluate_objective(x, number_of_samples,
+                    rvec_list, obj_list, num_samples_run, exit_info = control.evaluate_objective(x, number_of_samples,
                                                                                                params)
                     
                     if num_samples_run > 0:
@@ -514,8 +533,9 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
 
             # Evaluate new point
             x = control.model.as_absolute_coordinates(xnew)
+            ##print("x from xnew again", x)
             number_of_samples = max(nsamples(control.delta, control.rho, current_iter, nruns_so_far), 1)
-            rvec_list, f_list, num_samples_run, exit_info = control.evaluate_objective(x, number_of_samples, params)
+            rvec_list, obj_list, num_samples_run, exit_info = control.evaluate_objective(x, number_of_samples, params)
             if np.any(np.isnan(rvec_list)):
                 # Just exit without saving the current point
                 # We should be able to do a hard restart though, because it's unlikely 
@@ -535,7 +555,7 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
                 break  # quit
 
             # Estimate f in order to compute 'actual reduction'
-            ratio, exit_info = control.calculate_ratio(current_iter, rvec_list[:num_samples_run, :], d, gopt, H)
+            ratio, exit_info = control.calculate_ratio(control.model.xopt(abs_coordinates=True), current_iter, rvec_list[:num_samples_run, :], d, gopt, H)
             if exit_info is not None:
                 if exit_info.able_to_do_restart() and params("restarts.use_restarts") and params(
                         "restarts.use_soft_restarts"):
@@ -565,9 +585,9 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
                 diagnostic_info.update_slow_iter(-1)  # n/a, unless otherwise update
             if ratio < params("tr_radius.eta1"):  # ratio < 0.1
                 if finished_growing:
-                    control.delta = min(params("tr_radius.gamma_dec") * control.delta, dnorm)
+                    control.delta = min(params("tr_radius.gamma_dec") * control.delta, dnorm) / tau
                 else:
-                    control.delta = min(params("growing.gamma_dec") * control.delta, dnorm)  # different gamma_dec
+                    control.delta = min(params("growing.gamma_dec") * control.delta, dnorm) / tau  # different gamma_dec
                 if params("logging.save_diagnostic_info"):
                     diagnostic_info.update_iter_type(ITER_ACCEPTABLE_NO_GEOM if ratio > 0.0
                                                      else ITER_UNSUCCESSFUL_NO_GEOM)  # we flag geom update below
@@ -651,7 +671,7 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
                         break  # quit
 
                 # Update list of successful steps
-                this_step_was_not_improvement = control.model.fsave is not None and control.model.fopt() > control.model.fsave
+                this_step_was_not_improvement = control.model.objsave is not None and control.model.objopt() > control.model.objsave
                 succ_steps_not_improvement.pop()  # remove last item
                 succ_steps_not_improvement.insert(0, this_step_was_not_improvement)  # add at beginning
                 # Terminate (not restart) if all are True
@@ -828,10 +848,10 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
                 if do_logging:
                     module_logger.info("New rho = %g after %i function evaluations" % (control.rho, control.nf))
                     if control.n() < params("logging.n_to_print_whole_x_vector"):
-                        module_logger.debug("Best so far: f = %.15g at x = " % (control.model.fopt())
+                        module_logger.debug("Best so far: f = %.15g at x = " % (control.model.objopt())
                                       + str(control.model.xopt(abs_coordinates=True)))
                     else:
-                        module_logger.debug("Best so far: f = %.15g at x = [...]" % (control.model.fopt()))
+                        module_logger.debug("Best so far: f = %.15g at x = [...]" % (control.model.objopt()))
                 continue  # next iteration
             else:
                 # Quit on rho=rhoend
@@ -857,14 +877,14 @@ def solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfu
     # (end main loop)
 
     # Quit & return the important information
-    x, rvec, f, jacmin, nsamples = control.model.get_final_results()
+    x, rvec, obj, jacmin, nsamples = control.model.get_final_results()
     if do_logging:
         module_logger.debug("At return from DFO-LS, number of function evals = %i" % nf)
-        module_logger.debug("Smallest objective value = %.15g at x = " % f + str(x))
-    return x, rvec, f, jacmin, nsamples, control.nf, control.nx, nruns_so_far, exit_info, diagnostic_info
+        module_logger.debug("Smallest objective value = %.15g at x = " % obj + str(x))
+    return x, rvec, obj, jacmin, nsamples, control.nf, control.nx, nruns_so_far, exit_info, diagnostic_info
 
 
-def solve(objfun, x0, args=(), bounds=None, projections=[], npt=None, rhobeg=None, rhoend=1e-8, maxfun=None, nsamples=None, user_params=None,
+def solve(objfun, x0, h=None, lh=None, prox_uh=None, argsf=(), argsh=(), argsprox=(), bounds=None, projections=[], npt=None, rhobeg=None, rhoend=1e-8, maxfun=None, nsamples=None, user_params=None,
           objfun_has_noise=False, scaling_within_bounds=False, do_logging=True, print_progress=False):
     x0 = x0.astype(float)
     n = len(x0)
@@ -934,13 +954,21 @@ def solve(objfun, x0, args=(), bounds=None, projections=[], npt=None, rhobeg=Non
 
     exit_info = None
     # Input & parameter checks
+    if exit_info is None and h is not None:
+        if prox_uh is None:
+            exit_info = ExitInformation(EXIT_INPUT_ERROR, "Must provide prox_uh input if h is not None")
+        elif lh is None:
+            exit_info = ExitInformation(EXIT_INPUT_ERROR, "Must provide lh input if h is not None")
+        elif lh <= 0.0:
+            exit_info = ExitInformation(EXIT_INPUT_ERROR, "lh must be strictly positive")
+
     if exit_info is None and npt < n + 1:
         exit_info = ExitInformation(EXIT_INPUT_ERROR, "npt must be >= n+1 for linear models with inexact interpolation")
 
-    if exit_info is None and rhobeg < 0.0:
+    if exit_info is None and rhobeg <= 0.0:
         exit_info = ExitInformation(EXIT_INPUT_ERROR, "rhobeg must be strictly positive")
 
-    if exit_info is None and rhoend < 0.0:
+    if exit_info is None and rhoend <= 0.0:
         exit_info = ExitInformation(EXIT_INPUT_ERROR, "rhoend must be strictly positive")
 
     if exit_info is None and rhobeg <= rhoend:
@@ -1013,12 +1041,12 @@ def solve(objfun, x0, args=(), bounds=None, projections=[], npt=None, rhobeg=Non
             x0 = xp.copy()
 
     # Enforce lower & upper bounds on x0
-    idx = (x0 <= xl)
+    idx = (x0 < xl)
     if np.any(idx):
         warnings.warn("x0 below lower bound, adjusting", RuntimeWarning)
     x0[idx] = xl[idx]
 
-    idx = (x0 >= xu)
+    idx = (x0 > xu)
     if np.any(idx):
         warnings.warn("x0 above upper bound, adjusting", RuntimeWarning)
     x0[idx] = xu[idx]
@@ -1028,9 +1056,9 @@ def solve(objfun, x0, args=(), bounds=None, projections=[], npt=None, rhobeg=Non
     nruns = 0
     nf = 0
     nx = 0
-    xmin, rmin, fmin, jacmin, nsamples_min, nf, nx, nruns, exit_info, diagnostic_info = \
-        solve_main(objfun, x0, args, xl, xu, projections, npt, rhobeg, rhoend, maxfun, nruns, nf, nx, nsamples, params,
-                    diagnostic_info, scaling_changes, default_growing_method_set_by_user=default_growing_method_set_by_user,
+    xmin, rmin, objmin, jacmin, nsamples_min, nf, nx, nruns, exit_info, diagnostic_info = \
+        solve_main(objfun, x0, argsf, xl, xu, projections, npt, rhobeg, rhoend, maxfun, nruns, nf, nx, nsamples, params,
+                    diagnostic_info, scaling_changes, h, lh, argsh, prox_uh, argsprox, default_growing_method_set_by_user=default_growing_method_set_by_user,
                    do_logging=do_logging, print_progress=print_progress)
 
     # Hard restarts loop
@@ -1045,27 +1073,27 @@ def solve(objfun, x0, args=(), bounds=None, projections=[], npt=None, rhobeg=Non
 
         if do_logging:
             module_logger.info("Restarting from finish point (f = %g) after %g function evals; using rhobeg = %g and rhoend = %g"
-                     % (fmin, nf, rhobeg, rhoend))
+                     % (objmin, nf, rhobeg, rhoend))
         if params("restarts.hard.use_old_rk"):
-            xmin2, rmin2, fmin2, jacmin2, nsamples2, nf, nx, nruns, exit_info, diagnostic_info = \
-                solve_main(objfun, xmin, args, xl, xu, projections, npt, rhobeg, rhoend, maxfun, nruns, nf, nx, nsamples, params,
-                            diagnostic_info, scaling_changes, r0_avg_old=rmin, r0_nsamples_old=nsamples_min,
+            xmin2, rmin2, objmin2, jacmin2, nsamples2, nf, nx, nruns, exit_info, diagnostic_info = \
+                solve_main(objfun, xmin, argsf, xl, xu, projections, npt, rhobeg, rhoend, maxfun, nruns, nf, nx, nsamples, params,
+                            diagnostic_info, scaling_changes, h, lh, argsh, prox_uh, argsprox, r0_avg_old=rmin, r0_nsamples_old=nsamples_min,
                            do_logging=do_logging, print_progress=print_progress)
         else:
-            xmin2, rmin2, fmin2, jacmin2, nsamples2, nf, nx, nruns, exit_info, diagnostic_info = \
-                solve_main(objfun, xmin, args, xl, xu, projections, npt, rhobeg, rhoend, maxfun, nruns, nf, nx, nsamples, params,
-                           diagnostic_info, scaling_changes, do_logging=do_logging, print_progress=print_progress)
+            xmin2, rmin2, objmin2, jacmin2, nsamples2, nf, nx, nruns, exit_info, diagnostic_info = \
+                solve_main(objfun, xmin, argsf, xl, xu, projections, npt, rhobeg, rhoend, maxfun, nruns, nf, nx, nsamples, params,
+                           diagnostic_info, scaling_changes, h, lh, argsh, prox_uh, argsprox, do_logging=do_logging, print_progress=print_progress)
 
-        if fmin2 < fmin or np.isnan(fmin):
+        if objmin2 < objmin or np.isnan(objmin):
             if do_logging:
-                module_logger.info("Successful run with new f = %s compared to old f = %s" % (fmin2, fmin))
+                module_logger.info("Successful run with new f = %s compared to old f = %s" % (objmin2, objmin))
             last_successful_run = nruns
-            (xmin, rmin, fmin, nsamples_min) = (xmin2, rmin2, fmin2, nsamples2)
+            (xmin, rmin, objmin, nsamples_min) = (xmin2, rmin2, objmin2, nsamples2)
             if jacmin2 is not None:  # may be None if finished during setup phase, in which case just use old Jacobian
                 jacmin = jacmin2
         else:
             if do_logging:
-                module_logger.info("Unsuccessful run with new f = %s compared to old f = %s" % (fmin2, fmin))
+                module_logger.info("Unsuccessful run with new f = %s compared to old f = %s" % (objmin2, objmin))
 
     if nruns - last_successful_run >= params("restarts.max_unsuccessful_restarts"):
         exit_info = ExitInformation(EXIT_SUCCESS, "Reached maximum number of unsuccessful restarts")
@@ -1077,7 +1105,7 @@ def solve(objfun, x0, args=(), bounds=None, projections=[], npt=None, rhobeg=Non
     if scaling_changes is not None and jacmin is not None:
         for i in range(n):
             jacmin[:, i] = jacmin[:, i] / scaling_changes[1][i]
-    results = OptimResults(remove_scaling(xmin, scaling_changes), rmin, fmin, jacmin, nf, nx, nruns, exit_flag, exit_msg)
+    results = OptimResults(remove_scaling(xmin, scaling_changes), rmin, objmin, jacmin, nf, nx, nruns, exit_flag, exit_msg)
     if params("logging.save_diagnostic_info"):
         df = diagnostic_info.to_dataframe(with_xk=params("logging.save_xk"), with_rk=params("logging.save_rk"))
         results.diagnostic_info = df

dfols/trust_region.py

@@ -29,14 +29,14 @@ solves
     s.t.  lower <= x <= upper
           ||x-xbase|| <= Delta
 With this value, the variable d=x-xbase solves the problem
-    min_s  abs(c + g' * d)
+    min_d  abs(c + g' * d)
     s.t.   lower <= xbase + d <= upper
           ||d|| <= delta
 Again, we have a version of this for handling arbitrary constraints
 The call
     x = ctrsbox_geometry(xbase, c, g, projections, Delta)
 Solves
-    min_s  abs(c + g' * d)
+    min_d  abs(c + g' * d)
     s.t.   xbase + d is feasible w.r.t. the constraint set C
           ||d|| <= delta
 
@@ -70,7 +70,7 @@ alternative licensing.
 # Ensure compatibility with Python 2
 from __future__ import absolute_import, division, print_function, unicode_literals
 
-from math import sqrt
+from math import sqrt, ceil
 import numpy as np
 try:
     import trustregion
@@ -79,13 +79,93 @@ except ImportError:
     # Fall back to Python implementation
     USE_FORTRAN = False
 
-from .util import dykstra, pball, pbox, sumsq, model_value
+from .util import dykstra, pball, pbox, sumsq, model_value, remove_scaling
 
-__all__ = ['ctrsbox', 'ctrsbox_geometry', 'trsbox', 'trsbox_geometry']
+__all__ = ['ctrsbox_sfista', 'ctrsbox_pgd', 'ctrsbox_geometry', 'trsbox', 'trsbox_geometry']
 
 ZERO_THRESH = 1e-14
 
-def ctrsbox(xopt, g, H, projections, delta, d_max_iters=100, d_tol=1e-10, use_fortran=USE_FORTRAN):
+def ctrsbox_sfista(xopt, g, H, projections, delta, h, L_h, prox_uh, argsh=(), argsprox=(), func_tol=1e-3, max_iters=500, d_max_iters=100, d_tol=1e-10, use_fortran=USE_FORTRAN, scaling_changes=None, sfista_iters_scale=1.0):
+    n = xopt.size
+    assert xopt.shape == (n,), "xopt has wrong shape (should be vector)"
+    assert g.shape == (n,), "g and xopt have incompatible sizes"
+    assert len(H.shape) == 2, "H must be a matrix"
+    assert H.shape == (n,n), "H and xopt have incompatible sizes"
+    assert np.allclose(H, H.T), "H must be symmetric"
+    assert delta > 0.0, "delta must be strictly positive"
+
+    # Initialization
+    d = np.zeros(n) # start with zero vector
+    y = np.zeros(n)
+    t = 1
+    k_H = np.linalg.norm(H, 2)
+    crvmin = -1.0
+    
+    # Number of iterations & smoothing parameter, from Theorem 10.57 in 
+    #   [A. Beck. First-order methods in optimization, SIAM, 2017]
+    # We do not use the values of k and mu given in the theorem statement, but rather the intermediate
+    # results on p313 (K1 for number of iterations, and the immediate next line for mu)
+    # Note: in the book's notation, Gamma=delta^2, alpha=1, beta=L_h^2/2, Lf=k_H [alpha and beta from Thm 10.51]
+    try:
+        MAX_LOOP_ITERS = ceil(sfista_iters_scale * delta * (L_h+sqrt(L_h*L_h+2*k_H*func_tol)) / func_tol)
+        MAX_LOOP_ITERS = min(MAX_LOOP_ITERS, max_iters)
+    except ValueError:
+        MAX_LOOP_ITERS = max_iters
+    u =  2 * delta / (MAX_LOOP_ITERS * L_h) # smoothing parameter
+    # u = 2 * func_tol / (L_h ** 2 + L_h * sqrt(L_h ** 2 + 2 * k_H * func_tol))  # the above choice works better in practice
+
+    def gradient_Fu(xopt, g, H, u, prox_uh, d):
+    # Calculate gradient_Fu,
+    # where Fu(d) := g(d) + h_u(d) and h_u(d) is a 1/u-smooth approximation of h.
+    # We assume that h is globally Lipschitz continous with constant L_h,
+    # then we can let h_u(d) be the Moreau Envelope M_h_u(d) of h.
+        return g + H @ d + (xopt + d - prox_uh(remove_scaling(xopt + d, scaling_changes), u, *argsprox)) / u
+
+    # Lipschitz constant of gradient_Fu
+    l = k_H + 1 / u 
+
+    # trust region is a ball of radius delta around xopt
+    trproj = lambda w: pball(w, xopt, delta)
+
+    # combine trust region constraints with user-entered constraints
+    P = list(projections)  # make a copy of the projections list
+    P.append(trproj)
+    def proj(d0):
+        p = dykstra(P, xopt+d0, max_iter=d_max_iters, tol=d_tol)
+        # we want the step only, so we subtract xopt
+        # from the new point: proj(xk+d) - xk
+        return p - xopt
+
+    # general step
+    model_value_best = model_value(g, H, d, xopt, h, argsh, scaling_changes)
+    d_best = d.copy()
+    for k in range(MAX_LOOP_ITERS):
+        prev_d = d.copy()
+        prev_t = t
+        # gradient_Fu at y
+        g_Fu = gradient_Fu(xopt, g, H, u, prox_uh, d, *argsprox)
+
+        # main update step
+        d = proj(y - g_Fu / l)
+        new_model_value = model_value(g, H, d, xopt, h, argsh, scaling_changes)
+        if new_model_value < model_value_best:
+            d_best = d.copy()
+            model_value_best = new_model_value
+
+        # update true gradient
+        # gnew is the gradient of the smoothed function
+        gnew = gradient_Fu(xopt, g, H, u, prox_uh, d, *argsprox)
+
+        # update CRVMIN
+        crv = d.dot(H).dot(d)/sumsq(d) if sumsq(d) >= ZERO_THRESH else crvmin
+        crvmin = min(crvmin, crv) if crvmin != -1.0 else crv
+        
+        # momentum update
+        t = (1 + sqrt(1 + 4*t*t)) / 2
+        y = d + (prev_t - 1) * (d - prev_d) / t
+    return d, gnew, crvmin
+
+def ctrsbox_pgd(xopt, g, H, projections, delta, d_max_iters=100, d_tol=1e-10, use_fortran=USE_FORTRAN):
     n = xopt.size
     assert xopt.shape == (n,), "xopt has wrong shape (should be vector)"
     assert g.shape == (n,), "g and xopt have incompatible sizes"
@@ -151,7 +231,6 @@ def ctrsbox(xopt, g, H, projections, delta, d_max_iters=100, d_tol=1e-10, use_fo
 
     return d, gnew, crvmin
 
-
 def trsbox(xopt, g, H, sl, su, delta, use_fortran=USE_FORTRAN):
     if use_fortran:
         return trustregion.solve(g, H, delta,

dfols/util.py

@@ -31,7 +31,7 @@ import scipy.linalg as LA
 import sys
 
 
-__all__ = ['sumsq', 'eval_least_squares_objective', 'model_value', 'random_orthog_directions_within_bounds',
+__all__ = ['sumsq', 'eval_least_squares_with_regularisation', 'model_value', 'random_orthog_directions_within_bounds',
            'random_directions_within_bounds', 'apply_scaling', 'remove_scaling', 'pbox', 'pball', 'dykstra', 'qr_rank']
 
 module_logger = logging.getLogger(__name__) 
@@ -47,9 +47,9 @@ def sumsq(x):
     return np.dot(x, x)
 
 
-def eval_least_squares_objective(objfun, x, args=(), verbose=True, eval_num=0, pt_num=0, full_x_thresh=6, check_for_overflow=True):
+def eval_least_squares_with_regularisation(objfun, x, h=None, argsf=(), argsh=(), verbose=True, eval_num=0, pt_num=0, full_x_thresh=6, check_for_overflow=True):
     # Evaluate least squares function
-    fvec = objfun(x, *args)
+    fvec = objfun(x, *argsf)
 
     if check_for_overflow:
         try:
@@ -62,20 +62,31 @@ def eval_least_squares_objective(objfun, x, args=(), verbose=True, eval_num=0, p
     else:
         f = sumsq(fvec)
 
+    # objective = least-squares + regularisation
+    obj = f
+    if h is not None:
+        # Evaluate regularisation term
+        hvalue = h(x, *argsh)
+        obj = f + hvalue
+
     if verbose:
         if len(x) < full_x_thresh:
-            module_logger.info("Function eval %i at point %i has f = %.15g at x = " % (eval_num, pt_num, f) + str(x))
+            module_logger.info("Function eval %i at point %i has obj = %.15g at x = " % (eval_num, pt_num, obj) + str(x))
         else:
-            module_logger.info("Function eval %i at point %i has f = %.15g at x = [...]" % (eval_num, pt_num, f))
+            module_logger.info("Function eval %i at point %i has obj = %.15g at x = [...]" % (eval_num, pt_num, obj))
 
-    return fvec, f
+    return fvec, obj
 
 
-def model_value(g, H, s):
-    # Calculate model value (s^T * g + 0.5* s^T * H * s) = s^T * (gopt + 0.5 * H*s)
+def model_value(g, H, s, xopt=(), h=None,argsh=(), scaling_changes=None):
+    # Calculate model value (s^T * g + 0.5* s^T * H * s) + h(xopt + s) = s^T * (gopt + 0.5 * H*s) + h(xopt + s)
     assert g.shape == s.shape, "g and s have incompatible sizes"
     Hs = H.dot(s)
-    return np.dot(s, g + 0.5*Hs)
+    rtn = np.dot(s, g + 0.5*Hs)
+    if h is not None:
+        hvalue = h(remove_scaling(xopt+s, scaling_changes), *argsh)
+        rtn += hvalue
+    return rtn
 
 
 def get_scale(dirn, delta, lower, upper):

DFO_LS-1.4.1.dist-info/RECORD

@@ -1,14 +0,0 @@
-dfols/__init__.py,sha256=D-x5glfZFfJ8-bdjA-4k4JFTDu1Eylaz3EL4GSH28eI,1605
-dfols/controller.py,sha256=LSeHZoKaKUEYgB1_2subjKskHJ8mWccMbn-LOpxJ7LM,42769
-dfols/diagnostic_info.py,sha256=2kEUkL-MS4eDENUf1r2hOWsntP8OxMDKi_kyHmrC9V4,6081
-dfols/hessian.py,sha256=sExx4J4KoGwHItbthX2odosB2ONbQFvLdlcod7PIh4k,4262
-dfols/model.py,sha256=q70zuqocNtsaXzNjWHcTdrS209BdQt4uY0GNtp0qlI8,18809
-dfols/params.py,sha256=_Va1ybnQDIzWaXvImcSeH8xnNE_A2zpAfBgDG74sc5c,17557
-dfols/solver.py,sha256=IKg3xWPLYlOW_zuTc_-HY_3ZvdDEfkyxARerERUQHlU,61264
-dfols/trust_region.py,sha256=hRKQx0fpSxol7dLZO0yrT7O5IDptPPSnDvxKQNZ3r0M,24603
-dfols/util.py,sha256=ysdIHTkrkWwCRKuGffofehKl-t5dT3sD9dfy0muI4ZI,9852
-DFO_LS-1.4.1.dist-info/LICENSE.txt,sha256=jOtLnuWt7d5Hsx6XXB2QxzrSe2sWWh3NgMfFRetluQM,35147
-DFO_LS-1.4.1.dist-info/METADATA,sha256=RR6KhJi4Ae_1PES8Bpzqm3AYK2w12V-2MyDyjaCDe80,8552
-DFO_LS-1.4.1.dist-info/WHEEL,sha256=GJ7t_kWBFywbagK5eo9IoUwLW6oyOeTKmQ-9iHFVNxQ,92
-DFO_LS-1.4.1.dist-info/top_level.txt,sha256=UfxRhaDN8HQx2_l17KbrDrERJ90OCN7VKkDMpYYbRLU,6
-DFO_LS-1.4.1.dist-info/RECORD,,