DFO-LS 1.2__py3-none-any.whl → 1.4.1__py3-none-any.whl

{DFO_LS-1.2.dist-info → DFO_LS-1.4.1.dist-info}/METADATA

@@ -1,41 +1,53 @@
 Metadata-Version: 2.1
 Name: DFO-LS
-Version: 1.2
+Version: 1.4.1
 Summary: A flexible derivative-free solver for (bound constrained) nonlinear least-squares minimization
-Home-page: https://github.com/numericalalgorithmsgroup/dfols/
-Author: Lindon Roberts
-Author-email: lindon.roberts@maths.ox.ac.uk
-License: GNU GPL
-Download-URL: https://github.com/numericalalgorithmsgroup/dfols/archive/v1.2.tar.gz
-Keywords: mathematics derivative free optimization nonlinear least squares
-Platform: UNKNOWN
+Author-email: Lindon Roberts <lindon.roberts@sydney.edu.au>
+Maintainer-email: Lindon Roberts <lindon.roberts@sydney.edu.au>
+License: GPL-3.0-or-later
+Project-URL: Homepage, https://github.com/numericalalgorithmsgroup/dfols
+Project-URL: Download, https://github.com/numericalalgorithmsgroup/dfols/releases/
+Project-URL: Bug Tracker, https://github.com/numericalalgorithmsgroup/dfols/issues/
+Project-URL: Documentation, https://numericalalgorithmsgroup.github.io/dfols/
+Project-URL: Source Code, https://github.com/numericalalgorithmsgroup/dfols
+Keywords: mathematics,optimization,least squares,derivative free optimization,nonlinear least squares
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
 Classifier: Framework :: IPython
 Classifier: Framework :: Jupyter
-Classifier: Intended Audience :: Financial and Insurance Industry
 Classifier: Intended Audience :: Science/Research
-Classifier: License :: OSI Approved :: GNU General Public License (GPL)
+Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
 Classifier: Operating System :: MacOS
 Classifier: Operating System :: Microsoft :: Windows
-Classifier: Operating System :: POSIX
 Classifier: Operating System :: Unix
 Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 2
 Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Scientific/Engineering :: Mathematics
-Requires-Dist: numpy (>=1.11)
-Requires-Dist: scipy (>=0.18)
-Requires-Dist: pandas (>=0.17)
-Requires-Dist: trustregion (>=1.1)
+Requires-Python: >=3.9
+Description-Content-Type: text/x-rst
+License-File: LICENSE.txt
+Requires-Dist: setuptools
+Requires-Dist: numpy
+Requires-Dist: scipy >=1.11
+Requires-Dist: pandas
+Provides-Extra: dev
+Requires-Dist: pytest ; extra == 'dev'
+Requires-Dist: Sphinx ; extra == 'dev'
+Requires-Dist: sphinx-rtd-theme ; extra == 'dev'
+Provides-Extra: trustregion
+Requires-Dist: trustregion >=1.1 ; extra == 'trustregion'
 
 ===================================================
 DFO-LS: Derivative-Free Optimizer for Least-Squares
 ===================================================
 
-.. image::  https://travis-ci.org/numericalalgorithmsgroup/dfols.svg?branch=master
-   :target: https://travis-ci.org/numericalalgorithmsgroup/dfols
+.. image::  https://github.com/numericalalgorithmsgroup/dfols/actions/workflows/python_testing.yml/badge.svg
+   :target: https://github.com/numericalalgorithmsgroup/dfols/actions
    :alt: Build Status
 
 .. image::  https://img.shields.io/badge/License-GPL%20v3-blue.svg
@@ -49,6 +61,10 @@ DFO-LS: Derivative-Free Optimizer for Least-Squares
 .. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.2630426.svg
    :target: https://doi.org/10.5281/zenodo.2630426
    :alt: DOI:10.5281/zenodo.2630426
+   
+.. image:: https://static.pepy.tech/personalized-badge/dfo-ls?period=total&units=international_system&left_color=black&right_color=green&left_text=Downloads
+   :target: https://pepy.tech/project/dfo-ls
+   :alt: Total downloads
 
 DFO-LS is a flexible package for solving nonlinear least-squares minimization, without requiring derivatives of the objective. It is particularly useful when evaluations of the objective function are expensive and/or noisy. DFO-LS is more flexible version of `DFO-GN <https://github.com/numericalalgorithmsgroup/dfogn>`_.
 
@@ -66,36 +82,49 @@ If you use DFO-LS in a paper, please cite:
 
 Cartis, C., Fiala, J., Marteau, B. and Roberts, L., `Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers <https://doi.org/10.1145/3338517>`_, *ACM Transactions on Mathematical Software*, 45:3 (2019), pp. 32:1-32:41.
 
+If you use DFO-LS for problems with constraints, including bound constraints, please also cite:
+
+Hough, M. and Roberts, L., `Model-Based Derivative-Free Methods for Convex-Constrained Optimization <https://doi.org/10.1137/21M1460971>`_, *SIAM Journal on Optimization*, 21:4 (2022), pp. 2552-2579.
+
 Requirements
 ------------
 DFO-LS requires the following software to be installed:
 
-* Python 2.7 or Python 3 (http://www.python.org/)
-* Fortran compiler (e.g. `gfortran <https://gcc.gnu.org/wiki/GFortran>`_), required by the `trustregion <https://github.com/lindonroberts/trust-region>`_ package.
+* Python 3.9 or higher (http://www.python.org/)
 
 Additionally, the following python packages should be installed (these will be installed automatically if using *pip*, see `Installation using pip`_):
 
-* NumPy 1.11 or higher (http://www.numpy.org/)
-* SciPy 0.18 or higher (http://www.scipy.org/)
-* Pandas 0.17 or higher (http://pandas.pydata.org/)
+* NumPy (http://www.numpy.org/)
+* SciPy version 1.11 or higher (http://www.scipy.org/)
+* Pandas (http://pandas.pydata.org/)
+
+**Optional package:** DFO-LS versions 1.2 and higher also support the `trustregion <https://github.com/lindonroberts/trust-region>`_ package for fast trust-region subproblem solutions. To install this, make sure you have a Fortran compiler (e.g. `gfortran <https://gcc.gnu.org/wiki/GFortran>`_) and NumPy installed, then run :code:`pip install trustregion`. You do not have to have trustregion installed for DFO-LS to work, and it is not installed by default.
+
+Installation using conda
+------------------------
+DFO-LS can be directly installed in Anaconda environments using `conda-forge <https://anaconda.org/conda-forge/dfo-ls>`_:
+
+.. code-block:: bash
+
+    $ conda install -c conda-forge dfo-ls
 
 Installation using pip
 ----------------------
 For easy installation, use `pip <http://www.pip-installer.org/>`_ as root:
 
- .. code-block:: bash
+.. code-block:: bash
 
     $ [sudo] pip install DFO-LS
 
 or alternatively *easy_install*:
 
- .. code-block:: bash
+.. code-block:: bash
 
     $ [sudo] easy_install DFO-LS
 
 If you do not have root privileges or you want to install DFO-LS for your private use, you can use:
 
- .. code-block:: bash
+.. code-block:: bash
 
     $ pip install --user DFO-LS
 
@@ -103,7 +132,7 @@ which will install DFO-LS in your home directory.
 
 Note that if an older install of DFO-LS is present on your system you can use:
 
- .. code-block:: bash
+.. code-block:: bash
 
     $ [sudo] pip install --upgrade DFO-LS
 
@@ -132,7 +161,7 @@ If you do not have root privileges or you want to install DFO-LS for your privat
 
 instead.
 
-To upgrade DFO-LS to the latest version, navigate to the top-level directory (i.e. the one containing :code:`setup.py`) and rerun the installation using :code:`pip`, as above:
+To upgrade DFO-LS to the latest version, navigate to the top-level directory (i.e. the one containing :code:`pyproject.toml`) and rerun the installation using :code:`pip`, as above:
 
  .. code-block:: bash
 
@@ -141,11 +170,12 @@ To upgrade DFO-LS to the latest version, navigate to the top-level directory (i.
 
 Testing
 -------
-If you installed DFO-LS manually, you can test your installation by running:
+If you installed DFO-LS manually, you can test your installation using the pytest package:
 
  .. code-block:: bash
 
-    $ python setup.py test
+    $ pip install pytest
+    $ python -m pytest --pyargs dfols
 
 Alternatively, the HTML documentation provides some simple examples of how to run DFO-LS.
 
@@ -165,10 +195,8 @@ If DFO-LS was installed manually you have to remove the installed files by hand
 
 Bugs
 ----
-Please report any bugs using GitHub's issue tracker.
+Please report any bugs using `GitHub's issue tracker <https://github.com/numericalalgorithmsgroup/dfols/issues>`_.
 
 License
 -------
 This algorithm is released under the GNU GPL license. Please `contact NAG <http://www.nag.com/content/worldwide-contact-information>`_ for alternative licensing.
-
-

DFO_LS-1.4.1.dist-info/RECORD

@@ -0,0 +1,14 @@
+dfols/__init__.py,sha256=D-x5glfZFfJ8-bdjA-4k4JFTDu1Eylaz3EL4GSH28eI,1605
+dfols/controller.py,sha256=LSeHZoKaKUEYgB1_2subjKskHJ8mWccMbn-LOpxJ7LM,42769
+dfols/diagnostic_info.py,sha256=2kEUkL-MS4eDENUf1r2hOWsntP8OxMDKi_kyHmrC9V4,6081
+dfols/hessian.py,sha256=sExx4J4KoGwHItbthX2odosB2ONbQFvLdlcod7PIh4k,4262
+dfols/model.py,sha256=q70zuqocNtsaXzNjWHcTdrS209BdQt4uY0GNtp0qlI8,18809
+dfols/params.py,sha256=_Va1ybnQDIzWaXvImcSeH8xnNE_A2zpAfBgDG74sc5c,17557
+dfols/solver.py,sha256=IKg3xWPLYlOW_zuTc_-HY_3ZvdDEfkyxARerERUQHlU,61264
+dfols/trust_region.py,sha256=hRKQx0fpSxol7dLZO0yrT7O5IDptPPSnDvxKQNZ3r0M,24603
+dfols/util.py,sha256=ysdIHTkrkWwCRKuGffofehKl-t5dT3sD9dfy0muI4ZI,9852
+DFO_LS-1.4.1.dist-info/LICENSE.txt,sha256=jOtLnuWt7d5Hsx6XXB2QxzrSe2sWWh3NgMfFRetluQM,35147
+DFO_LS-1.4.1.dist-info/METADATA,sha256=RR6KhJi4Ae_1PES8Bpzqm3AYK2w12V-2MyDyjaCDe80,8552
+DFO_LS-1.4.1.dist-info/WHEEL,sha256=GJ7t_kWBFywbagK5eo9IoUwLW6oyOeTKmQ-9iHFVNxQ,92
+DFO_LS-1.4.1.dist-info/top_level.txt,sha256=UfxRhaDN8HQx2_l17KbrDrERJ90OCN7VKkDMpYYbRLU,6
+DFO_LS-1.4.1.dist-info/RECORD,,

{DFO_LS-1.2.dist-info → DFO_LS-1.4.1.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: bdist_wheel (0.34.2)
+Generator: bdist_wheel (0.43.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

dfols/__init__.py

@@ -7,8 +7,7 @@ nonlinear least-squares solver which only requires function values.
 
 It solves the nonlinear least-squares problem:
     min_{x}  f(x) = r1(x)**2 + ... + rm(x)**2,
-subject to the (optional) bounds
-    lb <= x <= ub,
+(optionally) subject to finitely many convex constraints,
 where each function ri(x) is differentiable, possibly nonconvex.
 Since the derivatives of ri(x) are never required or approximated,
 the solver works when the evaluation of ri(x) is noisy.
@@ -39,10 +38,10 @@ alternative licensing.
 # Ensure compatibility with Python 2
 from __future__ import absolute_import, division, print_function, unicode_literals
 
-from .version import __version__
-__all__ = ['__version__']
+# DFO-LS version
+__version__ = '1.4.1'
 
 # Main solver & exit flags
 from .solver import *
-__all__ += ['solve']
+__all__ = ['solve']
 

dfols/controller.py

@@ -41,8 +41,11 @@ from .util import *
 
 __all__ = ['Controller', 'ExitInformation', 'EXIT_SLOW_WARNING', 'EXIT_MAXFUN_WARNING', 'EXIT_SUCCESS',
            'EXIT_INPUT_ERROR', 'EXIT_TR_INCREASE_ERROR', 'EXIT_LINALG_ERROR', 'EXIT_FALSE_SUCCESS_WARNING',
-           'EXIT_AUTO_DETECT_RESTART_WARNING']
+           'EXIT_AUTO_DETECT_RESTART_WARNING', 'EXIT_EVAL_ERROR']
 
+module_logger = logging.getLogger(__name__) 
+
+EXIT_TR_INCREASE_WARNING = 5  # warning, TR increase in proj constrained case - likely due to multiple active constraints
 EXIT_AUTO_DETECT_RESTART_WARNING = 4  # warning, auto-detected restart criteria
 EXIT_FALSE_SUCCESS_WARNING = 3  # warning, maximum fake successful steps reached
 EXIT_SLOW_WARNING = 2  # warning, maximum number of slow (successful) iterations reached
@@ -51,6 +54,7 @@ EXIT_SUCCESS = 0  # successful finish (rho=rhoend, sufficient objective reductio
 EXIT_INPUT_ERROR = -1  # error, bad inputs
 EXIT_TR_INCREASE_ERROR = -2  # error, trust region step increased model value
 EXIT_LINALG_ERROR = -3  # error, linalg error (singular matrix encountered)
+EXIT_EVAL_ERROR = -4  # error, objective evaluation error (e.g. nan result received)
 
 
 class ExitInformation(object):
@@ -70,6 +74,8 @@ class ExitInformation(object):
             return "Warning (slow progress): " + self.msg
         elif self.flag == EXIT_MAXFUN_WARNING:
             return "Warning (max evals): " + self.msg
+        elif self.flag == EXIT_TR_INCREASE_WARNING:
+            return "Warning (trust region increase): " + self.msg
         elif self.flag == EXIT_INPUT_ERROR:
             return "Error (bad input): " + self.msg
         elif self.flag == EXIT_TR_INCREASE_ERROR:
@@ -78,11 +84,13 @@ class ExitInformation(object):
             return "Error (linear algebra): " + self.msg
         elif self.flag == EXIT_FALSE_SUCCESS_WARNING:
             return "Warning (max false good steps): " + self.msg
+        elif self.flag == EXIT_EVAL_ERROR:
+            return "Error (function evaluation): " + self.msg
         else:
             return "Unknown exit flag: " + self.msg
 
     def able_to_do_restart(self):
-        if self.flag in [EXIT_TR_INCREASE_ERROR, EXIT_LINALG_ERROR, EXIT_SLOW_WARNING, EXIT_AUTO_DETECT_RESTART_WARNING]:
+        if self.flag in [EXIT_TR_INCREASE_ERROR, EXIT_TR_INCREASE_WARNING, EXIT_LINALG_ERROR, EXIT_SLOW_WARNING, EXIT_AUTO_DETECT_RESTART_WARNING, EXIT_EVAL_ERROR]:
             return True
         elif self.flag in [EXIT_MAXFUN_WARNING, EXIT_INPUT_ERROR]:
             return False
@@ -92,13 +100,13 @@ class ExitInformation(object):
 
 
 class Controller(object):
-    def __init__(self, objfun, args, x0, r0, r0_nsamples, xl, xu, npt, rhobeg, rhoend, nf, nx, maxfun, params,
+    def __init__(self, objfun, args, x0, r0, r0_nsamples, xl, xu, projections, npt, rhobeg, rhoend, nf, nx, maxfun, params,
                  scaling_changes, do_logging):
         self.do_logging = do_logging
         self.objfun = objfun
         self.args = args
         self.maxfun = maxfun
-        self.model = Model(npt, x0, r0, xl, xu, r0_nsamples, precondition=params("interpolation.precondition"),
+        self.model = Model(npt, x0, r0, xl, xu, projections, r0_nsamples, precondition=params("interpolation.precondition"),
                            abs_tol = params("model.abs_tol"), rel_tol = params("model.rel_tol"), do_logging=do_logging)
         self.nf = nf
         self.nx = nx
@@ -107,9 +115,6 @@ class Controller(object):
         self.rho = rhobeg
         self.rhoend = rhoend
         self.diffs = [0.0, 0.0, 0.0]
-        self.last_iters_step_taken = []
-        self.last_fopts_step_taken = []
-        self.num_slow_iters = 0
         self.finished_growing = False
         self.finished_halfway_growing = False
         # For measuing slow iterations
@@ -134,12 +139,113 @@ class Controller(object):
 
     def initialise_coordinate_directions(self, number_of_samples, num_directions, params):
         if self.do_logging:
-            logging.debug("Initialising with coordinate directions")
+            module_logger.debug("Initialising with coordinate directions")
         # self.model already has x0 evaluated, so only need to initialise the other points
         # num_directions = params("growing.ndirs_initial")
         assert self.model.num_pts <= (self.n() + 1) * (self.n() + 2) // 2, "prelim: must have npt <= (n+1)(n+2)/2"
         assert 1 <= num_directions < self.model.num_pts, "Initialisation: must have 1 <= ndirs_initial < npt"
 
+
+        if self.model.projections:
+            D = np.zeros((self.n(),self.n()))
+            k = 0
+            while k < self.n():
+                ek = np.zeros(self.n())
+                ek[k] = 1
+                p = np.dot(ek,min(1,self.delta))
+                yk = dykstra(self.model.projections, self.model.xbase + p, max_iter=params("dykstra.max_iters"), tol=params("dykstra.d_tol"))
+                D[k,:] = yk - self.model.xbase
+
+                k += 1 # move on to next point
+
+            # Have at least one L.D. vector, try negative direction on bad one first
+            k = 0
+            mr_tol = params("matrix_rank.r_tol")
+            D_rank, diag = qr_rank(D,tol=mr_tol)
+            while D_rank != num_directions and k < self.n():
+                if diag[k] < mr_tol:
+                    ek = np.zeros(self.n())
+                    ek[k] = 1
+                    p = -np.dot(ek,min(1,self.delta))
+                    yk = dykstra(self.model.projections, self.model.xbase + p, max_iter=params("dykstra.max_iters"), tol=params("dykstra.d_tol"))
+                    dk = D[k,:].copy()
+                    D[k,:] = yk - self.model.xbase
+                    D_rank2, _diag2 = qr_rank(D,tol=params("matrix_rank.r_tol"))
+                    if D_rank2 <= D_rank:
+                        # Did not improve rank, revert change
+                        D[k,:] = dk
+                    # rank was improved, update D_rank for next comparison
+                    D_rank = D_rank2
+                k += 1
+
+            # Try random combination of negatives...
+            k = 0
+            slctr = np.random.randint(0, 1+1, self.n()) # generate rand binary "selector" array
+            D_rank, diag = qr_rank(D,tol=params("matrix_rank.r_tol"))
+            while D_rank != num_directions and k < 100*self.n():
+                if slctr[k%self.n()] == 1: # if selector says make -ve, make -ve
+                    ek = np.zeros(self.n())
+                    ek[k%self.n()] = 1
+                    p = -np.dot(ek,min(1,self.delta))
+                    yk = dykstra(self.model.projections, self.model.xbase + p, max_iter=params("dykstra.max_iters"), tol=params("dykstra.d_tol"))
+                    dk = D[k%self.n(),:].copy()
+                    D[k%self.n(),:] = yk - self.model.xbase
+                    D_rank2, _diag2 = qr_rank(D,tol=params("matrix_rank.r_tol"))
+                    if D_rank2 <= D_rank:
+                        # Did not improve rank, revert change
+                        D[k%self.n(),:] = dk
+                    # rank was improved, update D_rank for next comparison
+                    D_rank = D_rank2
+
+                # Go again
+                slctr = np.random.randint(0, 1+1, self.n())
+                k += 1
+
+            # Set still not L.I? Try random directions
+            i = 0
+            D_rank, diag = qr_rank(D,tol=params("matrix_rank.r_tol"))
+            while D_rank != num_directions and i <= 100*num_directions:
+                k = 0
+                while k < self.n():
+                    if diag[k] < mr_tol:
+                        p = np.random.normal(size=self.n())
+                        p = p/np.linalg.norm(p)
+                        p = np.dot(p,min(1,self.delta))
+                        yk = dykstra(self.model.projections, self.model.xbase + p, max_iter=params("dykstra.max_iters"), tol=params("dykstra.d_tol"))
+                        dk = D[k,:].copy()
+                        D[k,:] = yk - self.model.xbase
+                        D_rank2, _diag2 = qr_rank(D,tol=params("matrix_rank.r_tol"))
+                        if D_rank2 <= D_rank:
+                            # Did not improve rank, revert change
+                            D[k,:] = dk
+                        # rank was improved, update D_rank for next comparison
+                        D_rank = D_rank2
+                    k += 1
+                i += 1
+
+            if D_rank != num_directions:
+                raise RuntimeError("Unable to generate suitable initial directions")
+
+            # we have a L.I set of interpolation points
+            for k in range(0,self.n()):
+                # Evaluate objective at this new point
+                x = self.model.as_absolute_coordinates(D[k, :])
+                rvec_list, f_list, num_samples_run, exit_info = self.evaluate_objective(x, number_of_samples, params)
+
+                # Handle exit conditions (f < min obj value or maxfun reached)
+                if exit_info is not None:
+                    if num_samples_run > 0:
+                        self.model.save_point(x, np.mean(rvec_list[:num_samples_run, :], axis=0), num_samples_run,
+                                              x_in_abs_coords=True)
+                    return exit_info  # return & quit
+
+                # Otherwise, add new results (increments model.npt_so_far)
+                self.model.change_point(k+1, x - self.model.xbase, rvec_list[0, :])  # expect step, not absolute x
+                for i in range(1, num_samples_run):
+                    self.model.add_new_sample(k+1, rvec_extra=rvec_list[i, :])
+            
+            return None   # return & continue
+
         at_lower_boundary = (self.model.sl > -0.01 * self.delta)  # sl = xl - x0, should be -ve, actually < -rhobeg
         at_upper_boundary = (self.model.su < 0.01 * self.delta)  # su = xu - x0, should be +ve, actually > rhobeg
 
@@ -150,17 +256,19 @@ class Controller(object):
             # k = 2n+1, ..., (n+1)(n+2)/2 --> off-diagonal directions
             if 1 <= k < self.n() + 1:  # first step along coord directions
                 dirn = k - 1  # direction to move in (0,...,n-1)
-                stepa = self.delta if not at_upper_boundary[dirn] else -self.delta
+                stepa = self.delta if not at_upper_boundary[dirn] else -self.delta # take a +delta step if at lower, -delta if at upper
                 stepb = None
-                xpts_added[k, dirn] = stepa
+                xpts_added[k, dirn] = stepa # set new (relative) point to the step since we haven't done any moving, so relative point is all zeros.
 
             elif self.n() + 1 <= k < 2 * self.n() + 1:  # second step along coord directions
                 dirn = k - self.n() - 1  # direction to move in (0,...,n-1)
-                stepa = xpts_added[k - self.n(), dirn]
-                stepb = -self.delta
+                stepa = xpts_added[k - self.n(), dirn] # previous step
+                stepb = -self.delta # new step
                 if at_lower_boundary[dirn]:
+                    # if at lower boundary, set the second step to be +ve
                     stepb = min(2.0 * self.delta, self.model.su[dirn])  # su = xu - x0, should be +ve
                 if at_upper_boundary[dirn]:
+                    # if at upper boundary, set the second step to be -ve
                     stepb = max(-2.0 * self.delta, self.model.sl[dirn])  # sl = xl - x0, should be -ve
                 xpts_added[k, dirn] = stepb
 
@@ -208,7 +316,7 @@ class Controller(object):
 
     def initialise_random_directions(self, number_of_samples, num_directions, params):
         if self.do_logging:
-            logging.debug("Initialising with random orthogonal directions")
+            module_logger.debug("Initialising with random orthogonal directions")
         # self.model already has x0 evaluated, so only need to initialise the other points
         assert 1 <= num_directions < self.model.num_pts, "Initialisation: must have 1 <= ndirs_initial < npt"
 
@@ -328,20 +436,28 @@ class Controller(object):
 
         return dirn * (step_length / LA.norm(dirn))
 
-    def trust_region_step(self):
+    def trust_region_step(self, params):
         # Build model for full least squares objectives
         gopt, H = self.model.build_full_model()
-        d, gnew, crvmin = trsbox(self.model.xopt(), gopt, H, self.model.sl, self.model.su, self.delta)
+        if self.model.projections:
+            d, gnew, crvmin = ctrsbox(self.model.xopt(abs_coordinates=True), gopt, H, self.model.projections, self.delta, d_max_iters=params("dykstra.max_iters"), d_tol=params("dykstra.d_tol"))
+        else:
+            d, gnew, crvmin = trsbox(self.model.xopt(), gopt, H, self.model.sl, self.model.su, self.delta)
         return d, gopt, H, gnew, crvmin
 
     def geometry_step(self, knew, adelt, number_of_samples, params):
         if self.do_logging:
-            logging.debug("Running geometry-fixing step")
+            module_logger.debug("Running geometry-fixing step")
         try:
             c, g = self.model.lagrange_gradient(knew)
             # c = 1.0 if knew == self.model.kopt else 0.0  # based at xopt, just like d
-            # Solve problem: bounds are sl <= xnew <= su, and ||xnew-xopt|| <= adelt
-            xnew = trsbox_geometry(self.model.xopt(), c, g, np.minimum(self.model.sl, 0.0), np.maximum(self.model.su, 0.0), adelt)
+            if self.model.projections:
+                # Solve problem: use projection onto arbitrary constraints, and ||xnew-xopt|| <= adelt
+                step = ctrsbox_geometry(self.model.xopt(abs_coordinates=True), c, g, self.model.projections, adelt, d_max_iters=params("dykstra.max_iters"), d_tol=params("dykstra.d_tol"))
+                xnew = self.model.xopt() + step
+            else:
+                # Solve problem: bounds are sl <= xnew <= su, and ||xnew-xopt|| <= adelt
+                xnew = trsbox_geometry(self.model.xopt(), c, g, np.minimum(self.model.sl, 0.0), np.maximum(self.model.su, 0.0), adelt)
         except LA.LinAlgError:
             exit_info = ExitInformation(EXIT_LINALG_ERROR, "Singular matrix encountered in geometry step")
             return exit_info  # didn't fix geometry - return & quit
@@ -502,13 +618,16 @@ class Controller(object):
     def calculate_ratio(self, current_iter, rvec_list, d, gopt, H):
         exit_info = None
         f = sumsq(np.mean(rvec_list, axis=0))  # estimate actual objective value
-        pred_reduction = - model_value(gopt, H, d)
+        pred_reduction = - model_value(gopt, H, d) # negative of m since m(0) = 0
         actual_reduction = self.model.fopt() - f
         self.diffs = [abs(actual_reduction - pred_reduction), self.diffs[0], self.diffs[1]]
         if min(sqrt(sumsq(d)), self.delta) > self.rho:  # if ||d|| >= rho, successful!
             self.last_successful_iter = current_iter
         if pred_reduction < 0.0:
-            exit_info = ExitInformation(EXIT_TR_INCREASE_ERROR, "Trust region step gave model increase")
+            if len(self.model.projections) > 1: # if we are using multiple projections, only warn since likely due to constraint intersection
+                exit_info = ExitInformation(EXIT_TR_INCREASE_WARNING, "Either multiple constraints are active or trust region step gave model increase")
+            else:
+                exit_info = ExitInformation(EXIT_TR_INCREASE_ERROR, "Either rust region step gave model increase")
 
         ratio = actual_reduction / pred_reduction
         return ratio, exit_info
@@ -529,12 +648,12 @@ class Controller(object):
         if this_iter_slow:
             self.num_slow_iters += 1
             if self.do_logging:
-                logging.info("Slow iteration (%g consecutive so far, max allowed %g)"
+                module_logger.info("Slow iteration (%g consecutive so far, max allowed %g)"
                              % (self.num_slow_iters, params("slow.max_slow_iters")))
         else:
             self.num_slow_iters = 0
             if self.do_logging:
-                logging.debug("Non-slow iteration")
+                module_logger.debug("Non-slow iteration")
         return this_iter_slow, self.num_slow_iters >= params("slow.max_slow_iters")
 
     def soft_restart(self, number_of_samples, nruns_so_far, params, x_in_abs_coords_to_save=None, rvec_to_save=None,
@@ -563,12 +682,17 @@ class Controller(object):
                               self.model.nsamples[self.model.kopt], x_in_abs_coords=True)
 
         if self.do_logging:
-            logging.info("Soft restart [currently, f = %g after %g function evals]" % (self.model.fopt(), self.nf))
+            module_logger.info("Soft restart [currently, f = %g after %g function evals]" % (self.model.fopt(), self.nf))
         # Resetting method: reset delta and rho, then move the closest 'num_steps' points to xk to improve geometry
         # Note: closest points because we are suddenly increasing delta & rho, so we want to encourage spreading out points
         self.delta = self.rhobeg
         self.rho = self.rhobeg
         self.diffs = [0.0, 0.0, 0.0]
+        
+        # Forget history of slow iterations
+        self.last_iters_step_taken = []
+        self.last_fopts_step_taken = []
+        self.num_slow_iters = 0
 
         all_sq_dist = self.model.distances_to_xopt()[:self.model.npt()]
         closest_points = np.argsort(all_sq_dist)
@@ -615,7 +739,7 @@ class Controller(object):
                     self.model.add_new_sample(self.model.npt() - 1, rvec_extra=rvec_list[i, :])
 
             if self.do_logging:
-                logging.info("Soft restart: added %g new directions, npt is now %g" % (num_pts_to_add, self.model.npt()))
+                module_logger.info("Soft restart: added %g new directions, npt is now %g" % (num_pts_to_add, self.model.npt()))
 
         # Otherwise, we are doing a restart
         self.last_successful_iter = 0

dfols/hessian.py

@@ -39,7 +39,7 @@ class Hessian(object):
     def __init__(self, n, vals=None):
         self.n = n
         if vals is None:
-            self.hq = np.zeros((n * (n + 1) // 2,), dtype=np.float)
+            self.hq = np.zeros((n * (n + 1) // 2,), dtype=float)
         else:
             assert isinstance(vals, np.ndarray), "Can only set Hessian from NumPy array"
             assert len(vals.shape) in [1, 2], "Can only set Hessian from vector or matrix"

dfols/model.py

@@ -36,12 +36,15 @@ import numpy as np
 import scipy.linalg as LA
 
 from .trust_region import trsbox_geometry
-from .util import sumsq
+from .util import sumsq, dykstra
 
 __all__ = ['Model']
 
+module_logger = logging.getLogger(__name__) 
+
+
 class Model(object):
-    def __init__(self, npt, x0, r0, xl, xu, r0_nsamples, n=None, m=None, abs_tol=1e-12, rel_tol=1e-20, precondition=True,
+    def __init__(self, npt, x0, r0, xl, xu, projections, r0_nsamples, n=None, m=None, abs_tol=1e-12, rel_tol=1e-20, precondition=True,
                  do_logging=True):
         if n is None:
             n = len(x0)
@@ -63,6 +66,7 @@ class Model(object):
         self.xbase = x0.copy()
         self.sl = xl - self.xbase  # lower bound w.r.t. xbase (require xpt >= sl)
         self.su = xu - self.xbase  # upper bound w.r.t. xbase (require xpt <= su)
+        self.projections = projections
         self.points = np.zeros((npt, n))  # interpolation points w.r.t. xbase
 
         # Function values
@@ -71,7 +75,7 @@ class Model(object):
         self.fval = np.inf * np.ones((npt, ))  # overall objective value for each xpt
         self.fval[0] = sumsq(r0)
         self.kopt = 0  # index of current iterate (should be best value so far)
-        self.nsamples = np.zeros((npt,), dtype=np.int)  # number of samples used to evaluate objective at each point
+        self.nsamples = np.zeros((npt,), dtype=int)  # number of samples used to evaluate objective at each point
         self.nsamples[0] = r0_nsamples
         self.fbeg = self.fval[0]  # f(x0), saved to check for sufficient reduction
 
@@ -123,6 +127,8 @@ class Model(object):
             return np.minimum(np.maximum(self.sl, self.points[k, :].copy()), self.su)
         else:
             # Apply bounds and convert back to absolute coordinates
+            if self.projections:
+                return dykstra(self.projections, self.xbase + self.points[k,:])
             return self.xbase + np.minimum(np.maximum(self.sl, self.points[k, :]), self.su)
 
     def rvec(self, k):
@@ -133,8 +139,10 @@ class Model(object):
         assert 0 <= k < self.npt(), "Invalid index %g" % k
         return self.fval[k]
 
-    def as_absolute_coordinates(self, x):
+    def as_absolute_coordinates(self, x, full_dykstra=False):
         # If x were an interpolation point, get the absolute coordinates of x
+        if self.projections:
+            return dykstra(self.projections, self.xbase + x)
         return self.xbase + np.minimum(np.maximum(self.sl, x), self.su)
 
     def xpt_directions(self, include_kopt=True):
@@ -301,12 +309,12 @@ class Model(object):
                 return col_scale(LA.solve_triangular(self.R, Qb), self.right_scaling)
         else:
             if self.do_logging:
-                logging.warning("model.solve_geom_system not using factorisation")
+                module_logger.warning("model.solve_geom_system not using factorisation")
             W, left_scaling, right_scaling = self.interpolation_matrix()
             return col_scale(LA.lstsq(W, col_scale(rhs * left_scaling))[0], right_scaling)
 
     def interpolate_mini_models_svd(self, verbose=False, make_full_rank=False, min_sing_val=1e-6, sing_val_frac=1.0, max_jac_cond=1e8,
-                                    get_chg_J=False):
+                                    get_chg_J=False, throw_error_on_nans=False):
         W, left_scaling, right_scaling = self.interpolation_matrix()
         self.factorise_geom_system()
         ls_interp_cond_num = np.linalg.cond(W) if verbose else 0.0  # scipy.linalg does not have condition number!
@@ -327,12 +335,18 @@ class Model(object):
             self.model_jac = np.dot(self.model_jac, np.dot(Qhat, Qhat.T))
 
         rhs = self.fval_v[fval_row_idx, :]  # size npt * m
+        if np.any(np.isnan(rhs)) and throw_error_on_nans:
+            if self.do_logging:
+                module_logger.warning("model.interpolate_mini_models_svd: NaNs encountered in objective evaluations, raising error")
+            raise np.linalg.LinAlgError("NaN encountered in objective evaluations")
         try:
             dg = self.solve_geom_system(rhs)  # size (n+1)*m
         except LA.LinAlgError:
             return False, None, None, None, None  # flag error
         except ValueError:
             return False, None, None, None, None  # flag error (e.g. inf or NaN encountered)
+        if not np.all(np.isfinite(dg)):  # another check for inf or NaN
+            return False, None, None, None, None
         J_old = self.model_jac.copy()
         self.model_jac = dg[1:,:].T
         self.model_const = dg[0,:] - np.dot(self.model_jac, xopt)  # shift base to xbase