FitBenchmarking 1.2.0__tar.gz → 1.4.0__tar.gz

{fitbenchmarking-1.2.0 → fitbenchmarking-1.4.0}/PKG-INFO

@@ -1,18 +1,19 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: FitBenchmarking
-Version: 1.2.0
+Version: 1.4.0
 Summary: FitBenchmarking: A tool for comparing fitting software
 Project-URL: Homepage, https://fitbenchmarking.github.io
 Project-URL: Documentation, https://fitbenchmarking.readthedocs.io/en/stable
 Project-URL: Repository, https://github.com/fitbenchmarking/fitbenchmarking
 Project-URL: Release Notes, https://github.com/fitbenchmarking/fitbenchmarking/releases
 Project-URL: Issues, https://github.com/fitbenchmarking/fitbenchmarking/issues
-Author: FitBenchmarkign Team
+Author: FitBenchmarking Team
+License-File: LICENSE.txt
 Keywords: benchmark,comparison,fitting,least squares,optimisation
 Classifier: License :: OSI Approved :: BSD License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Requires-Python: >=3.9.1
+Requires-Python: >=3.10.1
 Requires-Dist: codecarbon<2.7,>=2.5.1
 Requires-Dist: configparser
 Requires-Dist: dash
@@ -22,11 +23,11 @@ Requires-Dist: lxml
 Requires-Dist: matplotlib>=2.0
 Requires-Dist: numpy
 Requires-Dist: pandas>=1.3
-Requires-Dist: plotly>=5.0.0
-Requires-Dist: scipy>=0.18
+Requires-Dist: plotly>=6.6.0
+Requires-Dist: scipy<1.16.0,>=0.18
 Requires-Dist: tqdm>=4.60
 Provides-Extra: bumps
-Requires-Dist: bumps>=0.9.0; extra == 'bumps'
+Requires-Dist: bumps<1.0.0; extra == 'bumps'
 Provides-Extra: dev
 Requires-Dist: coverage[toml]>=6.3; extra == 'dev'
 Requires-Dist: coveralls; extra == 'dev'
@@ -39,10 +40,12 @@ Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
 Requires-Dist: ruff; extra == 'dev'
 Provides-Extra: dfo
 Requires-Dist: dfo-ls; extra == 'dfo'
+Provides-Extra: galahad
+Requires-Dist: galahad-optrove; extra == 'galahad'
 Provides-Extra: gofit
 Requires-Dist: gofit; extra == 'gofit'
 Provides-Extra: gradient-free
-Requires-Dist: gradient-free-optimizers; extra == 'gradient-free'
+Requires-Dist: gradient-free-optimizers[sklearn]; extra == 'gradient-free'
 Provides-Extra: hogben
 Requires-Dist: hogben; extra == 'hogben'
 Provides-Extra: levmar
@@ -68,22 +71,22 @@ Requires-Dist: sasmodels; extra == 'sas'
 Requires-Dist: tinycc; (platform_system == 'Windows') and extra == 'sas'
 Description-Content-Type: text/markdown
 
-[![Build Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/release.yml?style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/release.yml?query=branch%3Av1.2.0)
-[![Tests Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/main.yml?label=tests&style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/main.yml?query=branch%3Av1.2.0)
+[![Build Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/release.yml?style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/release.yml?query=branch%3Av1.4.0)
+[![Tests Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/main.yml?label=tests&style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/main.yml?query=branch%3Av1.4.0)
 [![Install Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/install.yml?label=install&style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/install.yml)
-[![Documentation Status](https://img.shields.io/readthedocs/fitbenchmarking/v1.2.0?style=flat-square)](https://fitbenchmarking.readthedocs.io/en/v1.2.0)
+[![Documentation Status](https://img.shields.io/readthedocs/fitbenchmarking/v1.4.0?style=flat-square)](https://fitbenchmarking.readthedocs.io/en/v1.4.0)
 [![Coverage Status](https://img.shields.io/coveralls/github/fitbenchmarking/fitbenchmarking.svg?style=flat-square)](https://coveralls.io/github/fitbenchmarking/fitbenchmarking)
 [![Chat](https://img.shields.io/badge/chat-CompareFitMinimizers-lightgrey.svg?style=flat-square&logo=slack)](https://slack.com/)
-[![Zenodo](https://img.shields.io/badge/Zenodo-10.5281/zenodo.11198140-blue.svg?style=flat-square)](https://doi.org/10.5281/zenodo.11198140)
+[![Zenodo](https://img.shields.io/badge/Zenodo-10.5281/zenodo.15546207-blue.svg?style=flat-square)](https://zenodo.org/records/15546207)
 [![JOSS](https://img.shields.io/badge/JOSS-10.21105/joss.03127-brightgreen.svg?style=flat-square)](https://doi.org/10.21105/joss.03127)
 # FitBenchmarking
 
 FitBenchmarking is an open source tool for comparing different minimizers/fitting frameworks. FitBenchmarking is cross platform and we support Windows, Linux and Mac OS. For questions, feature requests or any other inquiries, please open an issue on GitHub.
 
-- **Installation Instructions:** https://fitbenchmarking.readthedocs.io/en/v1.2.0/users/install_instructions/index.html
-- **User Documentation & Example Usage:** https://fitbenchmarking.readthedocs.io/en/v1.2.0/users/index.html
-- **Community Guidelines:** https://fitbenchmarking.readthedocs.io/en/v1.2.0/contributors/guidelines.html
-- **Automated Tests:** Run via GitHub Actions, https://github.com/fitbenchmarking/fitbenchmarking/actions, and tests are documented at https://fitbenchmarking.readthedocs.io/en/v1.2.0/users/tests.html
+- **Installation Instructions:** https://fitbenchmarking.readthedocs.io/en/v1.4.0/users/install_instructions/index.html
+- **User Documentation & Example Usage:** https://fitbenchmarking.readthedocs.io/en/v1.4.0/users/index.html
+- **Community Guidelines:** https://fitbenchmarking.readthedocs.io/en/v1.4.0/contributors/guidelines.html
+- **Automated Tests:** Run via GitHub Actions, https://github.com/fitbenchmarking/fitbenchmarking/actions, and tests are documented at https://fitbenchmarking.readthedocs.io/en/v1.4.0/users/tests.html
 
 The package is the result of a collaboration between STFC’s Scientific Computing Department and ISIS Neutron and Muon Facility and the Diamond Light Source. We also would like to acknowledge support from:
 

{fitbenchmarking-1.2.0 → fitbenchmarking-1.4.0}/README.md

@@ -1,19 +1,19 @@
-[![Build Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/release.yml?style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/release.yml?query=branch%3Av1.2.0)
-[![Tests Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/main.yml?label=tests&style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/main.yml?query=branch%3Av1.2.0)
+[![Build Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/release.yml?style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/release.yml?query=branch%3Av1.4.0)
+[![Tests Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/main.yml?label=tests&style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/main.yml?query=branch%3Av1.4.0)
 [![Install Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/install.yml?label=install&style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/install.yml)
-[![Documentation Status](https://img.shields.io/readthedocs/fitbenchmarking/v1.2.0?style=flat-square)](https://fitbenchmarking.readthedocs.io/en/v1.2.0)
+[![Documentation Status](https://img.shields.io/readthedocs/fitbenchmarking/v1.4.0?style=flat-square)](https://fitbenchmarking.readthedocs.io/en/v1.4.0)
 [![Coverage Status](https://img.shields.io/coveralls/github/fitbenchmarking/fitbenchmarking.svg?style=flat-square)](https://coveralls.io/github/fitbenchmarking/fitbenchmarking)
 [![Chat](https://img.shields.io/badge/chat-CompareFitMinimizers-lightgrey.svg?style=flat-square&logo=slack)](https://slack.com/)
-[![Zenodo](https://img.shields.io/badge/Zenodo-10.5281/zenodo.11198140-blue.svg?style=flat-square)](https://doi.org/10.5281/zenodo.11198140)
+[![Zenodo](https://img.shields.io/badge/Zenodo-10.5281/zenodo.15546207-blue.svg?style=flat-square)](https://zenodo.org/records/15546207)
 [![JOSS](https://img.shields.io/badge/JOSS-10.21105/joss.03127-brightgreen.svg?style=flat-square)](https://doi.org/10.21105/joss.03127)
 # FitBenchmarking
 
 FitBenchmarking is an open source tool for comparing different minimizers/fitting frameworks. FitBenchmarking is cross platform and we support Windows, Linux and Mac OS. For questions, feature requests or any other inquiries, please open an issue on GitHub.
 
-- **Installation Instructions:** https://fitbenchmarking.readthedocs.io/en/v1.2.0/users/install_instructions/index.html
-- **User Documentation & Example Usage:** https://fitbenchmarking.readthedocs.io/en/v1.2.0/users/index.html
-- **Community Guidelines:** https://fitbenchmarking.readthedocs.io/en/v1.2.0/contributors/guidelines.html
-- **Automated Tests:** Run via GitHub Actions, https://github.com/fitbenchmarking/fitbenchmarking/actions, and tests are documented at https://fitbenchmarking.readthedocs.io/en/v1.2.0/users/tests.html
+- **Installation Instructions:** https://fitbenchmarking.readthedocs.io/en/v1.4.0/users/install_instructions/index.html
+- **User Documentation & Example Usage:** https://fitbenchmarking.readthedocs.io/en/v1.4.0/users/index.html
+- **Community Guidelines:** https://fitbenchmarking.readthedocs.io/en/v1.4.0/contributors/guidelines.html
+- **Automated Tests:** Run via GitHub Actions, https://github.com/fitbenchmarking/fitbenchmarking/actions, and tests are documented at https://fitbenchmarking.readthedocs.io/en/v1.4.0/users/tests.html
 
 The package is the result of a collaboration between STFC’s Scientific Computing Department and ISIS Neutron and Muon Facility and the Diamond Light Source. We also would like to acknowledge support from:
 

{fitbenchmarking-1.2.0 → fitbenchmarking-1.4.0}/fitbenchmarking/cli/checkpoint_handler.py

@@ -11,6 +11,7 @@ import os
 import sys
 import textwrap
 from argparse import ArgumentParser, RawDescriptionHelpFormatter
+from collections import defaultdict
 
 from fitbenchmarking.cli.exception_handler import exception_handler
 from fitbenchmarking.core.results_output import (
@@ -142,7 +143,7 @@ def generate_report(options_file="", additional_options=None, debug=False):
     :type options_file: str, optional
     :param additional_options: Extra options for the reporting.
                                Available keys are:
-                                   filename (str): The checkpoint file to use.
+                               filename (str): The checkpoint file to use.
     :type additional_options: dict, optional
     """
     if additional_options is None:
@@ -155,6 +156,20 @@ def generate_report(options_file="", additional_options=None, debug=False):
     checkpoint = Checkpoint(options=options)
     results, unselected_minimizers, failed_problems, config = checkpoint.load()
 
+    # Update options.software and options.minimizers
+    # so that they hold the correct values rather than
+    # the default. This update is necessary for processing
+    # the multstart plots.
+    set_minimizers = defaultdict(set)
+    for results_list in results.values():
+        for r in results_list:
+            set_minimizers[r.software].add(r.minimizer)
+    minimizers = defaultdict(
+        list, {k: list(v) for k, v in set_minimizers.items()}
+    )
+    options.software = list(minimizers.keys())
+    options.minimizers = minimizers
+
     all_dirs = []
     pp_dfs_all_prob_sets = {}
     for label in results:
@@ -185,46 +200,65 @@ def merge_data_sets(
 ):
     """
     Combine multiple checkpoint files into one following these rules:
-     1) The output will be the same as combining them one at a time in
-        sequence. i.e. A + B + C = (A + B) + C
-        The rules from here on will only reference A and B as the relation is
-        recursive.
-     2) Datasets
-         2a) Datasets in A and B are identical if they agree on the label
-         2b) If A and B contain identical datasets, the problems and results
-             are combined as below.
-             The remainder of these rules assume that A and B are identical
-             datasets as the alternative is trivial.
-     3) Problems
-         3a) If problems in A and B agree on name, ini_params, ini_y, x, y,
-             and e then these problems are considered identical.
-         3b) If A and B share identical problems, the details not specified in
-             3a are taken from A.
-         3c) If problems in A and B are not identical but share a name, the
-             name of the project in B should be updated to "<problem_name>*".
-     4) Results
-         4a) If results in A and B have identical problems and agree on name,
-             software_tag, minimizer_tag, jacobian_tag, hessian_tag, and
-             costfun_tag they are considered identical.
-         4b) If A an B share identical results, the details not specified in 4a
-             are taken from A if strategy is 'first', or B if strategy is
-             'last'.
-     5) As tables are grids of results, combining arbitrary results can lead to
-        un-table-able checkpoint files.
-        This occurs when the problems in A and B are not all identical and the
-        set of combinations of software_tag, minimizer_tag, jacobian_tag,
-        hessian_tag, and costfun_tag for which there are results in each of A
-        and B are not identical.
-        E.g. B has a problem not in A and uses a minimizer for which there are
-             no results in A.
-         5a) If the resulting checkpoint file would have the above issue, the
-             checkpoints are incompatible.
-         5b) Incompatible checkpoint files can be combined but should raise
-             warnings and mark the dataset in the checkpoint file.
-             Note: Some datasets may be incompatible where others can be
-                   successfully combined.
-     6) Unselected minimizers and failed problems will be discarded when
-        combining.
+
+    1) The output will be the same as combining them one at a time in
+       sequence. i.e. A + B + C = (A + B) + C
+
+        **Note:** The rules from here on will only reference A and B as
+        the relation is recursive.
+
+    2) Datasets
+
+        2a) Datasets in A and B are identical if they agree on the label
+
+        2b) If A and B contain identical datasets, the problems and results
+        are combined as below.
+
+        **Note:** The remainder of these rules assume that A and B are
+        identical datasets as the alternative is trivial.
+
+    3) Problems
+
+        3a) If problems in A and B agree on name, ini_params, ini_y, x, y,
+        and e then these problems are considered identical.
+
+        3b)If A and B share identical problems, the details not specified in
+        3a are taken from A.
+
+        3c) If problems in A and B are not identical but share a name, the
+        name of the project in B should be updated to "<problem_name>*".
+
+    4) Results
+
+        4a) If results in A and B have identical problems and agree on name,
+        software_tag, minimizer_tag, jacobian_tag, hessian_tag, and
+        costfun_tag they are considered identical.
+
+        4b) If A an B share identical results, the details not specified in 4a
+        are taken from A if strategy is 'first', or B if strategy is 'last'.
+
+    5) As tables are grids of results, combining arbitrary results can lead to
+       un-table-able checkpoint files.
+
+       This occurs when the problems in A and B are not all identical and the
+       set of combinations of software_tag, minimizer_tag, jacobian_tag,
+       hessian_tag, and costfun_tag for which there are results in each of A
+       and B are not identical.
+
+       **E.g.** B has a problem not in A and uses a minimizer for which there
+       are no results in A.
+
+            5a) If the resulting checkpoint file would have the above issue,
+            the checkpoints are incompatible.
+
+            5b) Incompatible checkpoint files can be combined but should raise
+            warnings and mark the dataset in the checkpoint file.
+
+        **Note:** Some datasets may be incompatible where others can be
+        successfully combined.
+
+    6) Unselected minimizers and failed problems will be discarded when
+       combining.
 
     :param files: The files to combine.
     :type files: list[str]

{fitbenchmarking-1.2.0 → fitbenchmarking-1.4.0}/fitbenchmarking/cli/main.py

@@ -122,8 +122,7 @@ of the Fitbenchmarking docs. """
         type=str,
         default=[],
         help=(
-            "Select what type of algorithm is used within a specific "
-            "software."
+            "Select what type of algorithm is used within a specific software."
         ),
     )
     parser.add_argument(
@@ -179,8 +178,16 @@ of the Fitbenchmarking docs. """
         help="Set the port for Dash.",
     )
 
-    group1 = parser.add_mutually_exclusive_group()
-    group1.add_argument(
+    parser.add_argument(
+        "--ip_address",
+        metavar="IP_ADDRESS",
+        type=str,
+        default="",
+        help="Set the ip address for Dash.",
+    )
+
+    make_plots_group = parser.add_mutually_exclusive_group()
+    make_plots_group.add_argument(
         "--make_plots",
         action="store_true",
         help=(
@@ -188,7 +195,7 @@ of the Fitbenchmarking docs. """
             "during runtime."
         ),
     )
-    group1.add_argument(
+    make_plots_group.add_argument(
         "--dont_make_plots",
         action="store_true",
         help=(
@@ -197,8 +204,8 @@ of the Fitbenchmarking docs. """
         ),
     )
 
-    group2 = parser.add_mutually_exclusive_group()
-    group2.add_argument(
+    results_browser_group = parser.add_mutually_exclusive_group()
+    results_browser_group.add_argument(
         "--results_browser",
         action="store_true",
         help=(
@@ -207,7 +214,7 @@ of the Fitbenchmarking docs. """
             "of a fit benchmark."
         ),
     )
-    group2.add_argument(
+    results_browser_group.add_argument(
         "--no_results_browser",
         action="store_true",
         help=(
@@ -217,8 +224,8 @@ of the Fitbenchmarking docs. """
         ),
     )
 
-    group3 = parser.add_mutually_exclusive_group()
-    group3.add_argument(
+    pbar_group = parser.add_mutually_exclusive_group()
+    pbar_group.add_argument(
         "--pbar",
         action="store_true",
         help=(
@@ -226,7 +233,7 @@ of the Fitbenchmarking docs. """
             "see the progress bar during runtime."
         ),
     )
-    group3.add_argument(
+    pbar_group.add_argument(
         "--no_pbar",
         action="store_true",
         help=(
@@ -269,8 +276,8 @@ of the Fitbenchmarking docs. """
         help="Specify the file path to write the logs to.",
     )
 
-    group2 = parser.add_mutually_exclusive_group()
-    group2.add_argument(
+    append_log_group = parser.add_mutually_exclusive_group()
+    append_log_group.add_argument(
         "--append_log",
         action="store_true",
         help=(
@@ -280,7 +287,7 @@ of the Fitbenchmarking docs. """
             "subsequent run."
         ),
     )
-    group2.add_argument(
+    append_log_group.add_argument(
         "--overwrite_log",
         action="store_true",
         help=(
@@ -297,7 +304,8 @@ of the Fitbenchmarking docs. """
         default="",
         help=(
             "Specify the minimum level of logging to display "
-            "on console during runtime."
+            "on console during runtime. Can be any of: "
+            "NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL ."
         ),
     )
     parser.add_argument(
@@ -305,7 +313,10 @@ of the Fitbenchmarking docs. """
         "--external_output",
         metavar="EXTERNAL_OUTPUT",
         default="",
-        help="Select the amount of information displayed from third-parties.",
+        help=(
+            "Select the amount of information displayed from third-parties. "
+            "Can be any of: display, log_only, debug."
+        ),
     )
 
     parser.add_argument(
@@ -313,13 +324,13 @@ of the Fitbenchmarking docs. """
         default=False,
         action="store_true",
         help=(
-            "Load results from the checkpoint and generate"
+            "Load results from the checkpoint and generate "
             "reports. Will not run any new tests."
         ),
     )
 
-    group4 = parser.add_mutually_exclusive_group()
-    group4.add_argument(
+    run_dash_group = parser.add_mutually_exclusive_group()
+    run_dash_group.add_argument(
         "--run_dash",
         action="store_true",
         help=(
@@ -327,7 +338,7 @@ of the Fitbenchmarking docs. """
             "interactive plots."
         ),
     )
-    group4.add_argument(
+    run_dash_group.add_argument(
         "--dont_run_dash",
         action="store_true",
         help=(
@@ -335,6 +346,23 @@ of the Fitbenchmarking docs. """
             "run dash for interactive plots."
         ),
     )
+
+    jacobian_check_group = parser.add_mutually_exclusive_group()
+    jacobian_check_group.add_argument(
+        "--check_jacobian",
+        action="store_true",
+        help=(
+            "Use this option to check the jacobian against "
+            "a finite difference approximation."
+        ),
+    )
+    jacobian_check_group.add_argument(
+        "--dont_check_jacobian",
+        action="store_true",
+        help=(
+            "Use this option if you have decided not to check the jacobian."
+        ),
+    )
     return parser
 
 
@@ -347,7 +375,7 @@ def run(problem_sets, additional_options=None, options_file="", debug=False):
     :param problem_sets: The paths to directories containing problem_sets.
     :type problem_sets: list of str
     :param additional_options: A dictionary of options input by the
-    user into the command line.
+                               user into the command line.
     :type additional_options: dict
     :param options_file: The path to an options file, defaults to ''.
     :type options_file: str, optional
@@ -383,76 +411,85 @@ def run(problem_sets, additional_options=None, options_file="", debug=False):
     pp_dfs_all_prob_sets = {}
     cp = Checkpoint(options=options)
 
-    for sub_dir in problem_sets:
-        # Create full path for the directory that holds a group of
-        # problem definition files
-        data_dir = os.path.join(current_path, sub_dir)
-
-        test_data = glob.glob(data_dir + "/*.*")
-
-        if not test_data:
-            LOGGER.warning("Problem set %s not found", data_dir)
-            continue
-
-        # generate group label/name used for problem set
-        try:
-            with open(
-                os.path.join(data_dir, "META.txt"), encoding="utf-8"
-            ) as f:
-                label = f.readline().strip("\n")
-        except OSError:
-            label = sub_dir.replace("/", "_")
-
-        LOGGER.info("Running the benchmarking on the %s problem set", label)
-        fit = Fit(
-            options=options, data_dir=data_dir, label=label, checkpointer=cp
-        )
-        results, failed_problems, unselected_minimizers = fit.benchmark()
-
-        # If a result has error flag 4 then the result contains dummy values,
-        # if this is the case for all results then output should not be
-        # produced as results tables won't show meaningful values.
-        all_dummy_results_flag = True
-        for result in results:
-            if result.error_flag != 4:
-                all_dummy_results_flag = False
-                break
-
-        # If the results are an empty list then this means that all minimizers
-        # raise an exception and the tables will produce errors if they run
-        # for that problem set.
-        if not results or all_dummy_results_flag:
-            message = (
-                "\nWARNING: \nThe user chosen options and/or problem "
-                " setup resulted in all minimizers and/or parsers "
-                "raising an exception. Because of this, results for "
-                f"the {label} problem set will not be displayed. "
-                "Please see the logs for more detail on why this is "
-                "the case."
+    try:
+        for sub_dir in problem_sets:
+            # Create full path for the directory that holds a group of
+            # problem definition files
+            data_dir = os.path.join(current_path, sub_dir)
+
+            test_data = glob.glob(data_dir + "/*.*")
+
+            if not test_data:
+                LOGGER.warning("Problem set %s not found", data_dir)
+                continue
+
+            # generate group label/name used for problem set
+            try:
+                with open(
+                    os.path.join(data_dir, "META.txt"), encoding="utf-8"
+                ) as f:
+                    label = f.readline().strip("\n")
+            except OSError:
+                label = sub_dir.replace("/", "_")
+
+            LOGGER.info(
+                "Running the benchmarking on the %s problem set", label
             )
-            LOGGER.warning(message)
-        else:
-            LOGGER.info("Producing output for the %s problem set", label)
-            # Display the runtime and accuracy results in a table
-            group_results_dir, pp_dfs = save_results(
-                group_name=label,
-                results=results,
+            fit = Fit(
                 options=options,
-                failed_problems=failed_problems,
-                unselected_minimizers=unselected_minimizers,
-                config=cp.config,
+                data_dir=data_dir,
+                label=label,
+                checkpointer=cp,
             )
-
-            pp_dfs_all_prob_sets[label] = pp_dfs
-
-            LOGGER.info("Completed benchmarking for %s problem set", sub_dir)
-            group_results_dir = os.path.relpath(
-                path=group_results_dir, start=options.results_dir
-            )
-            result_dir.append(group_results_dir)
-            group_labels.append(label)
-
-    cp.finalise()
+            results, failed_problems, unselected_minimizers = fit.benchmark()
+
+            # If a result has error flag 4 then the result contains dummy
+            # values, if this is the case for all results then output should
+            # not be produced as results tables won't show meaningful values.
+            all_dummy_results_flag = True
+            for result in results:
+                if result.error_flag != 4:
+                    all_dummy_results_flag = False
+                    break
+
+            # If the results are an empty list then this means that all
+            # minimizers raise an exception and the tables will produce
+            # errors if they run for that problem set.
+            if not results or all_dummy_results_flag:
+                message = (
+                    "\nWARNING: \nThe user chosen options and/or problem "
+                    " setup resulted in all minimizers and/or parsers "
+                    "raising an exception. Because of this, results for "
+                    f"the {label} problem set will not be displayed. "
+                    "Please see the logs for more detail on why this is "
+                    "the case."
+                )
+                LOGGER.warning(message)
+            else:
+                LOGGER.info("Producing output for the %s problem set", label)
+                # Display the runtime and accuracy results in a table
+                group_results_dir, pp_dfs = save_results(
+                    group_name=label,
+                    results=results,
+                    options=options,
+                    failed_problems=failed_problems,
+                    unselected_minimizers=unselected_minimizers,
+                    config=cp.config,
+                )
+
+                pp_dfs_all_prob_sets[label] = pp_dfs
+
+                LOGGER.info(
+                    "Completed benchmarking for %s problem set", sub_dir
+                )
+                group_results_dir = os.path.relpath(
+                    path=group_results_dir, start=options.results_dir
+                )
+                result_dir.append(group_results_dir)
+                group_labels.append(label)
+
+    finally:
+        cp.finalise()
 
     # Check result_dir is non empty before producing output
     if not result_dir:
@@ -516,6 +553,7 @@ def main():
         "run_name": args.run_name,
         "runtime_metric": args.runtime_metric,
         "port": args.port,
+        "ip_address": args.ip_address,
     }
 
     # Check if make_plots in options.py should be overridden, and if so,
@@ -539,6 +577,13 @@ def main():
     elif args.dont_run_dash:
         options_dictionary["run_dash"] = False
 
+    # Check if check_jacobian in options.py should be overridden, and if so,
+    # add to options_dictionary
+    if args.check_jacobian:
+        options_dictionary["check_jacobian"] = True
+    elif args.dont_check_jacobian:
+        options_dictionary["check_jacobian"] = False
+
     # Check if benchmark in options.py should be overridden, and if so,
     # add to options_dictionary
     if args.pbar: