pipstools 0.2.1__tar.gz

pipstools-0.2.1/PKG-INFO

@@ -0,0 +1,115 @@
+Metadata-Version: 2.4
+Name: pipstools
+Version: 0.2.1
+Summary: Add your description here
+Author: Manuel Wetzel
+Author-email: Manuel Wetzel <manuel.wetzel@dlr.de>
+License-Expression: MIT
+Classifier: Development Status :: 4 - Beta
+Classifier: Operating System :: Unix
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Requires-Dist: gamsapi
+Requires-Dist: gamspy-base
+Requires-Dist: highspy
+Requires-Dist: matplotlib
+Requires-Dist: mtkahypar ; sys_platform != 'win32'
+Requires-Dist: networkx
+Requires-Dist: numpy
+Requires-Dist: polars
+Requires-Dist: pyarrow
+Requires-Dist: scipy
+Requires-Dist: tqdm
+Requires-Dist: typer
+Requires-Dist: typing-extensions
+Requires-Dist: gurobipy ; extra == 'gurobi'
+Requires-Dist: mosek ; extra == 'mosek'
+Requires-Dist: coverage ; extra == 'test'
+Requires-Dist: pytest ; extra == 'test'
+Requires-Dist: pytest-cov ; extra == 'test'
+Requires-Python: >=3.11
+Provides-Extra: gurobi
+Provides-Extra: mosek
+Provides-Extra: test
+Description-Content-Type: text/markdown
+
+# PIPS-IPM++ tools
+
+`pipstools` is a collection of software tools that aims to give modellers easy access to the [massively parallel solver PIPS-IPM++](https://gitlab.com/pips-ipmpp).
+The tools can be used to identify suitable block structures for linear optimisation problems, either based on domain information or 
+in a fully automated manner using hypergraph partitioning. Furthermore, the identified block structures can be scored to estimate 
+their expected performance with PIPS-IPM++, and can be visually inspected to help modellers improve the problem formulation. 
+
+
+## Getting started
+
+If you are using [uv](https://docs.astral.sh/uv) as your python package manager, annotation a problem file can be done with a single call:
+
+```
+uvx git+https://git@gitlab.com/pips-ipmpp/pipstools annotate problem.mps
+```
+
+For solving the problem with PIPS-IPM++ using e.g. a Docker imange please refer to the [solver documentation](https://gitlab.com/pips-ipmpp/pips-ipmpp#docker-image).
+
+## Annotation
+
+Problem annotation describes the process of identifying underlying block strucutres which the massivly parallel solver PIPS-IPM++ can 
+exploit. Problem files can be provided either as .lp, .mps, or .gdx. If gurobipy is used as a file reader, also compressed 
+files (.mps.gz, .lp.gz) can be provided. `pipstools` currently support two differnt modes of identifying block strucutres, user-defined 
+annotations using domain knowledge and fully automated annotation using hypergraph partitioning.
+
+### Using variables names and regular expressions
+
+Typcially, users are aware of the strucutre and interdependencies between variables and equation of their optimization problem. This  can
+provide a good starting point by specifying variables naming patterns to be used for the decomposition. In the [sample file](/../main/samples/remix.mps.gz)
+of an energy system optimization model generated with the [REMix framework](https://gitlab.com/dlr-ve/esy/remix/framework) the temporal
+dimension uses the naming pattern of `tm` followed by the integer of the specific timestep, e.g. `tm1` to `tm8760` for each timestep of a year.
+This specific pattern can be encoded as the regular expression "tm([0-9]+)" which captures the integer representing the timestep to maintain
+chronological ordering. Note, that using regular expressions for the annotation requires the problem file to include the actual variables
+as generated with the specific mathematical programming tool used.
+
+```
+pipstools annotate samples/remix.mps --regex-pattern=tm([0-9]+)
+```
+
+### Using hypergraph partitioning
+For problems without variable names or prior knowledge about the underlying problem strucutre, a fully automated annotation can be used.
+The algorithm first splits any dense variables (automatically detected by heuristics, can be changed using the `--densecol` argument), which 
+in the case of energy system optimization problems typically corrosponds to investment decisions. The remaining problem strucute is then
+converted to a hypergraph using variables as nodes and constraints as hyperedges. Using the [Karlsruhe Hypergraph Partitioner](https://github.com/kahypar/mt-kahypar),
+a balanced partitioning into `n` partitions for the variables is generated (corresponding to the `--blocks n` argument). Afterwards, the
+constraints of the optimization problem are similarly paritioned taking into account the variable partions included in each constraint. 
+
+```
+pipstools annotate samples/remix.mps --method hypergraph --blocks 30
+```
+
+## Visualization
+
+After the identification of block structures using the annotation functionality, the identified block structure can be explored visually
+using the included visulisation tool. By default it will try to plot the matrix using a matplot display and include both the annotation 
+as well as variable and equation names. The matrix can be zoomed into, which further help refining the block structure and model formulation.
+If no display is available, the matrix plot can also be redirected to an image using the `--output` argument.
+
+```
+pipstools visualize samples/remix.gdx
+```
+
+## Scoring of block strucutres
+
+Scoring of the annotation is currently a work-in-progress feature.
+
+
+## Authors
+- Manuel Wetzel (German Aerospace Center, DLR)
+- Stephen Maher (GAMS Software GmbH)
+
+## Acknoledgements
+`pipstools` was developed as a deliverable of the PEREGRINE project, which was funded by 
+the German Federal Ministry for Economic Affairs and Energy under grant number 03EI1082A-B.

pipstools-0.2.1/README.md

@@ -0,0 +1,74 @@
+# PIPS-IPM++ tools
+
+`pipstools` is a collection of software tools that aims to give modellers easy access to the [massively parallel solver PIPS-IPM++](https://gitlab.com/pips-ipmpp).
+The tools can be used to identify suitable block structures for linear optimisation problems, either based on domain information or 
+in a fully automated manner using hypergraph partitioning. Furthermore, the identified block structures can be scored to estimate 
+their expected performance with PIPS-IPM++, and can be visually inspected to help modellers improve the problem formulation. 
+
+
+## Getting started
+
+If you are using [uv](https://docs.astral.sh/uv) as your python package manager, annotation a problem file can be done with a single call:
+
+```
+uvx git+https://git@gitlab.com/pips-ipmpp/pipstools annotate problem.mps
+```
+
+For solving the problem with PIPS-IPM++ using e.g. a Docker imange please refer to the [solver documentation](https://gitlab.com/pips-ipmpp/pips-ipmpp#docker-image).
+
+## Annotation
+
+Problem annotation describes the process of identifying underlying block strucutres which the massivly parallel solver PIPS-IPM++ can 
+exploit. Problem files can be provided either as .lp, .mps, or .gdx. If gurobipy is used as a file reader, also compressed 
+files (.mps.gz, .lp.gz) can be provided. `pipstools` currently support two differnt modes of identifying block strucutres, user-defined 
+annotations using domain knowledge and fully automated annotation using hypergraph partitioning.
+
+### Using variables names and regular expressions
+
+Typcially, users are aware of the strucutre and interdependencies between variables and equation of their optimization problem. This  can
+provide a good starting point by specifying variables naming patterns to be used for the decomposition. In the [sample file](/../main/samples/remix.mps.gz)
+of an energy system optimization model generated with the [REMix framework](https://gitlab.com/dlr-ve/esy/remix/framework) the temporal
+dimension uses the naming pattern of `tm` followed by the integer of the specific timestep, e.g. `tm1` to `tm8760` for each timestep of a year.
+This specific pattern can be encoded as the regular expression "tm([0-9]+)" which captures the integer representing the timestep to maintain
+chronological ordering. Note, that using regular expressions for the annotation requires the problem file to include the actual variables
+as generated with the specific mathematical programming tool used.
+
+```
+pipstools annotate samples/remix.mps --regex-pattern=tm([0-9]+)
+```
+
+### Using hypergraph partitioning
+For problems without variable names or prior knowledge about the underlying problem strucutre, a fully automated annotation can be used.
+The algorithm first splits any dense variables (automatically detected by heuristics, can be changed using the `--densecol` argument), which 
+in the case of energy system optimization problems typically corrosponds to investment decisions. The remaining problem strucute is then
+converted to a hypergraph using variables as nodes and constraints as hyperedges. Using the [Karlsruhe Hypergraph Partitioner](https://github.com/kahypar/mt-kahypar),
+a balanced partitioning into `n` partitions for the variables is generated (corresponding to the `--blocks n` argument). Afterwards, the
+constraints of the optimization problem are similarly paritioned taking into account the variable partions included in each constraint. 
+
+```
+pipstools annotate samples/remix.mps --method hypergraph --blocks 30
+```
+
+## Visualization
+
+After the identification of block structures using the annotation functionality, the identified block structure can be explored visually
+using the included visulisation tool. By default it will try to plot the matrix using a matplot display and include both the annotation 
+as well as variable and equation names. The matrix can be zoomed into, which further help refining the block structure and model formulation.
+If no display is available, the matrix plot can also be redirected to an image using the `--output` argument.
+
+```
+pipstools visualize samples/remix.gdx
+```
+
+## Scoring of block strucutres
+
+Scoring of the annotation is currently a work-in-progress feature.
+
+
+## Authors
+- Manuel Wetzel (German Aerospace Center, DLR)
+- Stephen Maher (GAMS Software GmbH)
+
+## Acknoledgements
+`pipstools` was developed as a deliverable of the PEREGRINE project, which was funded by 
+the German Federal Ministry for Economic Affairs and Energy under grant number 03EI1082A-B.

pipstools-0.2.1/pyproject.toml

@@ -0,0 +1,74 @@
+[project]
+name = "pipstools"
+version = "0.2.1"
+description = "Add your description here"
+readme = "README.md"
+license = "MIT"
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Operating System :: Unix",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering",
+]
+requires-python = ">=3.11"
+dependencies = [
+    "gamsapi",
+    "gamspy_base",
+    "highspy",
+    "matplotlib",
+    "mtkahypar ; platform_system != 'Windows'",
+    "networkx",
+    "numpy",
+    "polars",
+    "pyarrow",
+    "scipy",
+    "tqdm",
+    "typer",
+    "typing-extensions",
+]
+
+[[project.authors]]
+name = "Manuel Wetzel"
+email = "manuel.wetzel@dlr.de"
+
+[project.optional-dependencies]
+test = [
+    "coverage",
+    "pytest",
+    "pytest-cov",
+]
+gurobi = ["gurobipy"]
+mosek = ["mosek"]
+
+[project.scripts]
+pipstools = "pipstools.cli:app"
+
+[build-system]
+requires = ["uv_build>=0.12.7,<0.13.0"]
+build-backend = "uv_build"
+
+[tool.uv]
+package = true
+
+[tool.ruff]
+line-length = 100
+fix = true
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "I",
+]
+ignore = ["E501"]
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+docstring-code-format = true

pipstools-0.2.1/pyproject.toml.orig

@@ -0,0 +1,65 @@
+[project]
+name = "pipstools"
+version = "0.2.1"
+description = "Add your description here"
+readme = "README.md"
+license = "MIT"
+authors = [{ name = "Manuel Wetzel", email = "manuel.wetzel@dlr.de" }]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Operating System :: Unix",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering",
+]
+
+requires-python = ">=3.11"
+dependencies = [
+    "gamsapi",
+    "gamspy_base",
+    "highspy",
+    "matplotlib",
+    "mtkahypar ; platform_system != 'Windows'",
+    "networkx",
+    "numpy",
+    "polars",
+    "pyarrow",
+    "scipy",
+    "tqdm",
+    "typer",
+    "typing-extensions",
+]
+
+[project.optional-dependencies]
+test = ["coverage", "pytest", "pytest-cov"]
+
+gurobi = ["gurobipy"]
+mosek = ["mosek"]
+
+[project.scripts]
+pipstools = "pipstools.cli:app"
+
+[build-system]
+requires = ["uv_build>=0.12.7,<0.13.0"]
+build-backend = "uv_build"
+
+[tool.uv]
+package = true
+
+[tool.ruff]
+line-length = 100
+fix = true
+
+[tool.ruff.lint]
+select = ["E", "F", "I"]
+ignore = ["E501"]
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+docstring-code-format = true

pipstools-0.2.1/src/pipstools/__init__.py

@@ -0,0 +1,5 @@
+from .annotation import annotate_problem as annotate_problem
+from .conversion import convert_problem as convert_problem
+from .scoring import score_problem as score_problem
+from .validation import validate_problem as validate_problem
+from .visualisation import visualize_problem as visualize_problem

pipstools-0.2.1/src/pipstools/annotation.py

@@ -0,0 +1,204 @@
+import gzip
+import importlib.util
+import shutil
+from pathlib import Path
+from time import perf_counter
+
+from pipstools.io import read_gdx, read_mps, read_parquet, write_gdx, write_parquet
+from pipstools.partitioning import get_partitions
+from pipstools.scoring import get_score
+from pipstools.utils import get_stem
+
+
+def annotate_problem(
+    f_input,
+    k,
+    f_output=None,
+    fileformat="gdx",
+    var_dense=None,
+    equ_dense=None,
+    presolve=False,
+    mpsreader=None,
+    method="hypergraph",
+    hypergraph="col",
+    hg_objective="soed",
+    regex_pattern=None,
+    regex_order=None,
+    regex_mapfile=None,
+    dec=None,
+    write_names=True,
+    write_uels=True,
+    distributed=False,
+    vcycles=0,
+):
+    start = perf_counter()
+    if isinstance(f_input, str):
+        f_input = Path(f_input)
+
+    if var_dense is not None and var_dense < 0:
+        var_dense = None
+
+    if equ_dense is not None and equ_dense < 0:
+        var_dense = None
+
+    if mpsreader is None:
+        lib_gurobi = importlib.util.find_spec("gurobipy")
+        if lib_gurobi is not None:
+            mpsreader = "gurobi"
+        else:
+            mpsreader = "highs"
+        print(f"Selected {mpsreader} as default mps reader.")
+
+    if k == 1 and method.lower() != "oneblock":
+        print('Specificed partitioning into a single block, switching method to "oneblock"')
+        method = "oneblock"
+
+    if regex_pattern is not None:
+        print('Specificed regex pattern, switching method to "regex"')
+        method = "regex"
+
+    if dec is not None:
+        print('Specificed .dec file, switching method to "dec"')
+        method = "dec"
+
+    # Check if valid hypergraph arguments are provided
+    if method == "hypergraph":
+        if hypergraph.lower() not in ["col", "row", "colrow", "rowcol"]:
+            raise IOError(
+                f"Hypergraph type {hypergraph} is not available. "
+                "Use either 'col', 'row', 'colrow', or 'rowcol'"
+            )
+        if hg_objective.lower() not in ["cut", "km1", "soed"]:
+            raise IOError(
+                f"Hypergraph objective {hg_objective} is not available. "
+                "Use either 'cut', 'km1', or 'soed'"
+            )
+
+    # Read in file and extract A, vars, cols
+    match f_input.suffix.lower():
+        case ".gdx":
+            A, cols, rows, objcol, objrow, objcoef, objjacval = read_gdx(f_input=f_input)
+        case ".parquet":
+            A, cols, rows, objcol, objrow, objcoef, objjacval = read_parquet(f_input=f_input)
+        case ".lp" | ".mps" | ".gz" | ".bz2":
+            compressed_input = f_input.suffix.lower() in [".gz", ".bz2"]
+
+            # File needs to be compresses lp or mps
+            if compressed_input:
+                suffix_uncomp = f_input.stem.lower()
+                if not suffix_uncomp.endswith(".lp") and not suffix_uncomp.endswith(".mps"):
+                    raise (
+                        IOError(
+                            "Can only read compressed files of type .lp and .mps, "
+                            f"not {f_input.suffix}"
+                        )
+                    )
+
+            # For highspy manually decompress file and delete later
+            remove_decompressed = False
+            if compressed_input and mpsreader == "highs":
+                print(f"Automatically decompressing problem file {f_input.as_posix()} for highs")
+                if f_input.with_suffix("").exists():
+                    raise (
+                        IOError(
+                            f"Cannot automatically decompress file {f_input.as_posix()}, since "
+                            "a file with the target name already exists!"
+                        )
+                    )
+                with gzip.open(f_input, "rb") as f_in:
+                    with open(f_input.with_suffix(""), "wb") as f_out:
+                        shutil.copyfileobj(f_in, f_out)
+                f_input = f_input.with_suffix("")
+                remove_decompressed = True
+
+            # Read in mps content
+            A, cols, rows, objcol, objrow, objcoef = read_mps(
+                f_input=f_input,
+                presolve=presolve,
+                read_names=write_names or method == "regex",
+                mpsreader=mpsreader,
+            )
+            objjacval = 1.0
+
+            # Remove temporary decompressed file again
+            if remove_decompressed:
+                print(f"Automatically removing decompressed file {f_input.as_posix()}")
+                f_input.unlink()
+        case _:
+            raise (IOError(f"No reader implemented for filetype {f_input.suffix}"))
+
+    # Check if kahypar is available
+    if method == "hypergraph":
+        lib_mkkahypar = importlib.util.find_spec("mtkahypar")
+        if lib_mkkahypar is None:
+            raise ImportError(
+                "Hypergraph partitioning specified, but library mtkahypar not available."
+            )
+
+    # Partition by regex, hypergraph or dense elements
+    cols, rows = get_partitions(
+        A=A,
+        cols=cols,
+        rows=rows,
+        method=method,
+        hypergraph=hypergraph,
+        hg_objective=hg_objective,
+        k=k,
+        var_dense=var_dense,
+        equ_dense=equ_dense,
+        regex_pattern=regex_pattern,
+        regex_order=regex_order,
+        regex_mapfile=regex_mapfile,
+        dec=dec,
+        vcycles=vcycles,
+        objcol=objcol,
+    )
+
+    # Derive stem for output
+    if f_output is None:
+        stem, suffix = get_stem(f_input)
+    else:
+        stem, suffix = get_stem(f_output)
+        if suffix is not None:
+            fileformat = suffix.lstrip(".")
+
+    f_output = Path(f"{stem}_{len(set(cols['partition'].unique()) - {1})}b")
+    f_output.parent.mkdir(parents=True, exist_ok=True)
+
+    # Get scoring for annotation
+    get_score(A, cols, rows)
+
+    # Write out annotated gdx file
+    match fileformat.lower():
+        case "gdx":
+            write_gdx(
+                f_input=f_input,
+                f_output=f_output,
+                A=A,
+                cols=cols,
+                rows=rows,
+                objcol=objcol,
+                objrow=objrow,
+                objcoef=objcoef,
+                objjacval=objjacval,
+                write_names=write_names,
+                write_uels=write_uels,
+                distributed=distributed,
+            )
+        case "parquet":
+            write_parquet(
+                f_input=f_input,
+                f_output=f_output,
+                A=A,
+                cols=cols,
+                rows=rows,
+                objcol=objcol,
+                objrow=objrow,
+                objcoef=objcoef,
+                objjacval=objjacval,
+            )
+        case _:
+            raise (IOError(f"No method implemented for output format {fileformat}"))
+
+    stop = perf_counter()
+    print(f"Annotated problem in {(stop - start):.2f} seconds")