potassco-benchmark-tool 2.1.1__tar.gz → 2.2.0__tar.gz

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: potassco-benchmark-tool
-Version: 2.1.1
+Version: 2.2.0
 Summary: A benchmark-tool for ASP based systems.
 Author: Roland Kaminski, Tom Schmidt
 License: MIT License
@@ -34,7 +34,7 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: lxml
 Requires-Dist: pandas
-Requires-Dist: odswriter
+Requires-Dist: xlsxwriter
 Requires-Dist: pyarrow
 Requires-Dist: nbformat
 Provides-Extra: format
@@ -67,16 +67,29 @@ A tool to easier generate, run and evaluate benchmarks.
 
 ## Installation
 
-The `setuptools` package is required to run the commands below. We recommend
-the usage of conda, which already includes `setuptools` in its default python
-installation. Any python version newer than 3.10 is supported.
+The benchmark tool can be installed with any Python version newer than 3.10
+using pip:
 
 ```bash
-$ git clone https://github.com/potassco/benchmark-tool
-$ cd benchmark-tool
-$ conda create -n <env-name> python=3.10
-$ conda activate <env-name>
-$ pip install .
+pip install potassco-benchmark-tool
+```
+
+To access the latest updates and fixes you can either use:
+
+```bash
+pip install git+https://github.com/potassco/benchmark-tool
+```
+
+Or alternatively build the tool yourself, which requires the `setuptools`
+package. We recommend using conda, which includes `setuptools` in its default
+Python installation. To build the tool manually run the following commands:
+
+```bash
+git clone https://github.com/potassco/benchmark-tool
+cd benchmark-tool
+conda create -n <env-name> python=3.10
+conda activate <env-name>
+pip install .
 ```
 
 The documentation can be accessed [here](https://potassco.org/benchmark-tool/)
@@ -104,7 +117,7 @@ Supported subcommands in order of use:
 - `run-dist` Run distributed jobs
 - `verify` Check for runlim errors and re-run failed instances
 - `eval` Collect results
-- `conv` Convert results to ODS or other formats
+- `conv` Convert results to spreadsheet and more
 
 For more information and examples check the documentation.
 

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/README.md

@@ -4,16 +4,29 @@ A tool to easier generate, run and evaluate benchmarks.
 
 ## Installation
 
-The `setuptools` package is required to run the commands below. We recommend
-the usage of conda, which already includes `setuptools` in its default python
-installation. Any python version newer than 3.10 is supported.
+The benchmark tool can be installed with any Python version newer than 3.10
+using pip:
 
 ```bash
-$ git clone https://github.com/potassco/benchmark-tool
-$ cd benchmark-tool
-$ conda create -n <env-name> python=3.10
-$ conda activate <env-name>
-$ pip install .
+pip install potassco-benchmark-tool
+```
+
+To access the latest updates and fixes you can either use:
+
+```bash
+pip install git+https://github.com/potassco/benchmark-tool
+```
+
+Or alternatively build the tool yourself, which requires the `setuptools`
+package. We recommend using conda, which includes `setuptools` in its default
+Python installation. To build the tool manually run the following commands:
+
+```bash
+git clone https://github.com/potassco/benchmark-tool
+cd benchmark-tool
+conda create -n <env-name> python=3.10
+conda activate <env-name>
+pip install .
 ```
 
 The documentation can be accessed [here](https://potassco.org/benchmark-tool/)
@@ -41,7 +54,7 @@ Supported subcommands in order of use:
 - `run-dist` Run distributed jobs
 - `verify` Check for runlim errors and re-run failed instances
 - `eval` Collect results
-- `conv` Convert results to ODS or other formats
+- `conv` Convert results to spreadsheet and more
 
 For more information and examples check the documentation.
 

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/docs/examples/index.md

@@ -7,29 +7,32 @@ hide:
 
 ## Sequential Benchmark
 
-The example assumes that you want to run a benchmark that shall be started using simple bash scripts. All the following instruction assume that the current working directory is the root directory of the benchmark-tool project. To begin, the two executables [clasp-3.4.0][1] and [runlim][2] have to be copied (or symlinked) into the `./programs` folder.  
+The example assumes that you want to run a benchmark that shall be started using simple bash scripts. To begin, call `btool init` and copy (or symlink) the two executables [clasp-3.4.0][1] and [runlim][2]
+into the `./programs` folder.  
 Now, run:  
 `$ btool gen ./runscripts/runscript-seq.xml`  
 This creates a set of start scripts in the `./output` folder.  
 To start the benchmark, run:  
 `$ ./output/clasp-big/houat/start.py`  
 Once the benchmark is finished, run:  
-`$ btool eval ./runscripts/runscript-seq.xml | btool conv -o result.ods`  
-Finally, open the file:  
-`$ soffice result.ods`  
+`$ btool eval ./runscripts/runscript-seq.xml | btool conv -o result.xlsx`  
+Finally, open the file in your favourite spreadsheet tool:  
+`$ xdg-open result.xlsx`  
 
 ## Cluster Benchmark
 
-This example assumes that you want to run a benchmark on a cluster, i.g. on the [HPC][3] cluster at the university of Potsdam. Again, all the following instruction assume that the current working directory is the root directory of the benchmark-tool project. Once again make sure, the two executables [clasp-3.4.0][1] and [runlim][2] have been copied (or symlinked) into the `./programs` folder.  
+This example assumes that you want to run a benchmark on a cluster. Once again,
+call `btool init` and make sure, the two executables [clasp-3.4.0][1]
+and [runlim][2] have been copied (or symlinked) into the `./programs` folder.  
 Now, run:  
 `$ btool gen ./runscripts/runscript-dist.xml`  
 This creates a set of start scripts in the `./output` folder.  
 To start the benchmark, run (on the cluster):  
 `$ ./output/clasp-one-as/hpc/start.sh`  
 Once the benchmark is finished, run:  
-`$ btool eval ./runscripts/runscript-dist.xml | btool conv -o result.ods`  
-Finally, open the file:  
-`$ soffice result.ods`  
+`$ btool eval ./runscripts/runscript-dist.xml | btool conv -o result.xlsx`  
+Finally, open the file in your favourite spreadsheet tool:  
+`$ xdg-open result.xlsx`  
 
 ## Runscripts
 This tool comes with a [collection](https://github.com/potassco/benchmark-tool/blob/master/runscripts) of example runscripts to help you get started.

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/docs/getting_started/conv/index.md

@@ -3,28 +3,36 @@ title: "Converting Benchmark Results to Spreadsheets"
 icon: "material/play-outline"
 ---
 
-The `conv` subcommand allows you to convert results generated by `btool eval` into an ODS
-spreadsheet, which can be opened with LibreOffice, OpenOffice, or Excel.
+The `conv` subcommand allows you to convert results generated by `btool eval` into an XLSX
+spreadsheet, which should be used with Excel. Other programs, such as LibreOffice
+or OpenOffice, should also work. Keep in mind that by default LibreOffice does not automatically
+recalculate formulas. Therefore, after opening the spreadsheet, use `CTRL + SHIFT + F9` to
+manually recalculate all formulas. You can also enable automatic recalculation in the settings.
 
 To convert your benchmark results to a spreadsheet, use the following command:
 
 ```bash
-btool conv benchmark-results.xml -m "time:t,choices" -o results.ods
+btool conv benchmark-results.xml -m "time:t,choices" -o results.xlsx
 ```
 
-The name of the resulting `.ods` file is set using the `-o, --output` option (default `out.ods`).
+The name of the resulting `.xlsx` file is set using the `-o, --output` option (default `out.xlsx`).
+In cases of a missing or wrong file extension the correct file extension is added.
 
 Which benchmark projects to include in the output can be selected via the `-p, --project`
 option. By default all projects are selected.
 
 The `-m, --measures` option specifies which measures to include in the table (default: time:t,timeout:to;
-`-m all` selects all measures). Available measures depend on the [result parser] used during evaluation. Each measure can optionally include a formatting argument after a `:`. Currently,
+`-m all` selects all measures). Available measures depend on the [result parser] used during evaluation.
+Each measure can optionally include a formatting argument after a `:`. Currently,
 the supported formatting options are `t` and `to`. Both highlight best and worst values for
 float measures. Use `t` for most measures, and `to` for float measures representing booleans,
 such as `timeout`.
 
+The `--max-col-width` option can be used to set the maximum column width of the spreadsheet
+in terms of pixel. The default is 300.
+
 You can chose to export the instance data to a `.parquet` file using the `-e, --export`
-option. The name of the file will be the same as the specified output, i.e. `-o res.ods -e`
+option. The name of the file will be the same as the specified output, i.e. `-o res.xlsx -e`
 -> `res.parquet`.
 
 The `-j, --jupyter-notebook` option can be used to generate a `.ipynb` file, which contains
@@ -34,7 +42,7 @@ the `-e` option.
 
 ## Spreadsheet Generation
 
-When generating a spreadsheet in ODS format, two sheets are created:
+When generating a spreadsheet in XLSX format, two sheets are created:
 
 1. **Instance Sheet**
     - The instance sheet lists all runs for each benchmark instance (rows) and
@@ -49,11 +57,11 @@ When generating a spreadsheet in ODS format, two sheets are created:
     comparisons between classes.
 
 !!! info
-    All summaries are written as formulas in the ODS file. The calculated
+    All summaries are written as formulas in the .xlsx file. The calculated
     values are also accessible via the *content* attribute of the *Sheet*
     object.
 
-    Both the ODS representation and the actual content are stored in [pandas]
+    Both the XLSX representation and the actual content are stored in [pandas]
     DataFrames for easier handling and future modifications.
 
 [result parser]: ../../reference/resultparser.md

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/docs/getting_started/gen/index.md

@@ -17,6 +17,9 @@ btool gen ./runscripts/runscript-example.xml
 You can use the `-e, --exclude` option to exclude previously finished benchmarks
 in the start script, thus avoiding running them again.
 
+If the output directory, specified in the runscript, already exists, the program
+is interrupted. The `-f, --force` option can be used to disable this behaviour
+and overwrite existing files.
 
 After generation, start your benchmarks by executing either the `start.sh` or
 `start.py` file found in the `machine` subfolder of the generated structure.

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/docs/getting_started/gen/runscript.md

@@ -103,6 +103,8 @@ benchmark result evaluation. This does not affect benchmark script generation.
 system.
 - The `cmdline` attribute is optional and can be any string, which will be passed
 to the system regardless of the setting.
+- The `cmdline_post` attribute is similar but is placed after `setting.cmdline`
+in the order of arguments.
 
 A runscript can contain any number of systems, each with any number of
 settings.
@@ -120,7 +122,9 @@ encodings used by the system.
 ```
 
 - The `cmdline` attribute can be any valid string, which will be passed to the
-system via the run template when this setting is selected.
+system after `system.cmdline` when this setting is selected.
+- The `cmdline_post` attribute is similar but is placed after `system.cmdline_post`
+in the order of arguments.
 - The `tag` attribute is a space seperated identifier used within the runscript
 to select multiple settings at once.
 - Each setting can contain any number of encoding elements.
@@ -130,24 +134,31 @@ to select multiple settings at once.
     instances when this setting is selected.
     - If a `tag` is given, encoding usage is instance-dependent. Multiple
     encodings can be selected by using the same tag.
-- The setting element also supports an optional `disttemplate` attribute. The
+- The setting element also supports an optional `dist_template` attribute. The
 default value is `templates/single.dist`, which refers to [single.dist]. This
 attribute is only relevant for distributed jobs. More information about dist
 templates can be found on the [templates] page.
-- Another optional attribute for distributed jobs is `distopts`, which allows
-    you to add additional options for distributed jobs. `distopts` expects a
-    comma-separated string of options. For example, `distopts="#SBATCH
-    --hint=compute_bound,#SBATCH --job-name=\"my_benchmark_run\""` results in
+- Another optional attribute for distributed jobs is `dist_options`, which allows
+    you to add additional options for distributed jobs. `dist_options` expects a
+    comma-separated string of options. For example,  
+    `dist_options="#SBATCH --hint=compute_bound,#SBATCH -J=%x.%j.out"` results in
     the following lines being added to the script:
 
     ```bash
     #SBATCH --hint=compute_bound
-    #SBATCH --job-name="my_benchmark_run"
+    #SBATCH -J=%x.%j.out
     ```
 
     The default template for distributed jobs uses SLURM; a comprehensive list
     of available options is provided in the [SLURM documentation].
 
+To summarize, the commandline arguments will always be given to the
+system-under-test in the following order:
+
+```
+system.cmdline setting.cmdline system.cmdline_post setting.cmdline_post
+```
+
 ## Job
 
 A job defines additional arguments for individual runs. You can define any
@@ -160,19 +171,22 @@ A sequential job is identified by its `name` and sets the `timeout` (in
 seconds) for a single run, the number of `runs` for each instance, and
 the number of solver processes executed in `parallel`. The optional
 attribute `memout` sets a memory limit (in MB) for each run. If no limit
-is set, a default limit of 2000 MB is used:
+is set, a default limit of 20 GB is used. Additional options, which will be
+passed to the runlim call, can be set using the optional `template_options` attribute.
+`template_options` expects a comma-separated string of options, e.g.  
+`template_options="--single,--report-rate=2000"`.
 
 ```xml
-<seqjob name="seq-gen" timeout="900" runs="1" memout="1000" parallel="1"/>
+<seqjob name="seq-gen" timeout="900" runs="1" memout="1000" template_options="--single" parallel="1"/>
 ```
 
 ### Distributed Jobs
 
 A distributed job is also identified by its `name` and defines a `timeout`,
-the number of `runs` and an optional `memout`:
+the number of `runs` and an optional `memout` and `template_options`:
 
 ```xml
-<distjob name="dist-gen" timeout="900" runs="1" memout="1000"
+<distjob name="dist-gen" timeout="900" runs="1" memout="1000" template_options="--single"
         script_mode="timeout" walltime="23h 59m 59s" cpt="4"/>
 ```
 

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/docs/getting_started/gen/templates.md

@@ -12,18 +12,19 @@ repository].
 ## Run Templates
 
 Run templates define how each benchmark instance is executed. During script
-generation, references within the template (e.g., `run.files`) are replaced
+generation, references within the template (e.g., `{files}`) are replaced
 with corresponding values.
 
 The following references are available:
 
-- `run.files`: instance files
-- `run.encodings`: encoding files used for this instance
-- `run.root`: path to the benchmark-tool folder
-- `run.timeout`: walltime for this run
-- `run.memout`: memory limit for this run in MB (default: 20000)
-- `run.solver`: solver or program used for this run
-- `run.args`: additional arguments for the solver/program
+- `files`: instance files
+- `encodings`: encoding files used for this instance
+- `root`: path to the benchmark-tool folder
+- `timeout`: walltime for this run
+- `memout`: memory limit for this run in MB (default: 20000)
+- `solver`: solver or program used for this run
+- `args`: additional arguments for the solver/program
+- `options`: additional options
 
 Most templates use the [runlim] program to supervise benchmark runs.
 

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/docs/getting_started/index.md

@@ -8,10 +8,15 @@ The benchmark tool can be installed with any Python version newer than 3.10 usin
 pip install potassco-benchmark-tool
 ```
 
-To access the latest updates and fixes you can alternatively
-build the tool yourself, which requires the `setuptools` package.
-We recommend using conda, which includes `setuptools` in its default
-Python installation. To build the tool manually run the following commands:
+To access the latest updates and fixes you can either use:
+
+```bash
+pip install git+https://github.com/potassco/benchmark-tool
+```
+
+Or alternatively build the tool yourself, which requires the `setuptools` package.
+We recommend using conda, which includes `setuptools` in its default Python
+installation. To build the tool manually run the following commands:
 
 ```bash
 git clone https://github.com/potassco/benchmark-tool
@@ -40,7 +45,7 @@ Supported subcommands in order of use:
 - `run-dist`: Run distributed jobs
 - `verify`: Check for runlim errors
 - `eval`: Collect results
-- `conv`: Convert results to ODS or other formats
+- `conv`: Convert results to spreadsheet and more
 
 
 Each subcommand has their own help page, which you can access using:

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/docs/getting_started/init/index.md

@@ -4,14 +4,14 @@ icon: "material/play-outline"
 ---
 
 The `init` subcommand can be used to prepare the necessary folder structure to run
-the benchmarktool and provide some example [runscripts] and script [templates]. By
-default existing files are not overwritten.
+the benchmarktool and provide some example [runscripts] and script [templates].
 
 ```bash
 btool init
 ```
 
-The `-o, --overwrite` option can be used to overwrite existing files.
+By default existing files are not overwritten. This can be changed using
+the `-f, --force` option.
 
 You can use the `--resultparser-template` option to create a copy of the `clasp` resultparser
 called `rp_tmp.py`, which you can use as a base to create your own. You can overwrite the

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/docs/getting_started/workflow/index.md

@@ -57,16 +57,15 @@ To evaluate your benchmarks and collect the results use the [eval] subcommand:
 btool eval <runscript.xml> > <results.xml>
 ```
 
-This newly created .xml file can then be used as input for the [conv] subcommand to generate an .ods file
-and optionally an .ipynb jupyter notebook. By default only the time and timeout measures are displayed.
-Further measures can be selected using the -m option.
+This newly created .xml file can then be used as input for the [conv] subcommand to generate an .xlsx
+file and optionally an .ipynb jupyter notebook. By default only the time and timeout measures are displayed. Further measures can be selected using the -m option.
 
 ```
-btool conv -o <out.ods> <result.xml>
+btool conv -o <out.xlsx> <result.xml>
 ```
 
-[runscripts]: ./gen/runscript.md
-[templates]: ./gen/templates.md
+[runscripts]: ../gen/runscript.md
+[templates]: ../gen/templates.md
 [dispatcher]: ../run_dist/index.md
 [verify]: ../verify/index.md
 [init]: ../init/index.md

potassco_benchmark_tool-2.1.1/docs/reference/api/result/ods_gen.md → potassco_benchmark_tool-2.2.0/docs/reference/api/result/xlsx_gen.md

@@ -1,10 +1,10 @@
 ---
-title: "ODS Gen"
+title: "XLSX Gen"
 ---
 
 # ODS Gen
 
-::: benchmarktool.result.ods_gen
+::: benchmarktool.result.xlsx_gen
     handler: python
     options:
       filters: public

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/mkdocs.yml

@@ -152,7 +152,7 @@ nav:
         - reference/api/result/index.md
         - Classes: reference/api/result/result.md
         - Parser: reference/api/result/parser.md
-        - ODS Gen: reference/api/result/ods_gen.md
+        - XLSX Gen: reference/api/result/xlsx_gen.md
       - Tools: reference/api/tools.md
       - Entry Points: reference/api/entry_points.md
   - Community:

{potassco_benchmark_tool-2.1.1 → potassco_benchmark_tool-2.2.0}/pyproject.toml

@@ -11,7 +11,7 @@ dynamic = ["version"]
 dependencies = [
     "lxml",
     "pandas",
-    "odswriter",
+    "xlsxwriter",
     "pyarrow",
     "nbformat"
 ]
@@ -84,7 +84,7 @@ variable-rgx = "^[a-z][a-z0-9]*((_[a-z0-9]+)*_?)?$"
 good-names = ["_"]
 
 [tool.pylint."messages control"]
-disable = ["consider-using-f-string", "duplicate-code"]
+disable = ["duplicate-code"]
 extension-pkg-allow-list = ["lxml"]
 
 [tool.coverage.run]