mdpsolver 0.9.1__tar.gz

mdpsolver-0.9.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Anders Reenberg Andersen and Jesper Fink Andersen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mdpsolver-0.9.1/MANIFEST.in

@@ -0,0 +1,3 @@
+graft src
+
+global-exclude __pycache__

mdpsolver-0.9.1/PKG-INFO

@@ -0,0 +1,146 @@
+Metadata-Version: 2.1
+Name: mdpsolver
+Version: 0.9.1
+Summary: An efficient solver for Markov Decision Processes
+Author-email: Anders Reenberg Andersen <andersra@live.dk>, Jesper Fink Andersen <jfan@dtu.dk>
+Project-URL: Homepage, https://github.com/areenberg/MDPSolver
+Project-URL: Issues, https://github.com/areenberg/MDPSolver/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# `mdpsolver`: An efficient solver for Markov Decision Processes
+ 
+`mdpsolver` is the Python package for deriving the policies of Markov Decision Processes (MDPs) with infinite-horizon and the expected total discounted reward optimality criterion. 
+
+## Features
+
+* Available on PyPI.
+* Solver-engine developed in C++.
+* Available optimization algorithms: *Value iteration*, *Policy iteration*, and *Modified policy iteration*.  
+* Includes a variety of input formats available for users to choose from.
+
+# Quick start guide
+
+The following shows how to get quickly started with `mdpsolver`.
+
+## Installation
+
+Download and install `mdpsolver` directly from PyPI. 
+
+```
+pip install mdpsolver
+```
+
+## Usage
+
+Start by specifying the reward function and transition probabilities as lists. The following is an example of a simple MDP containing **three states** and **two actions in each state**.
+
+```python
+#Import packages
+import mdpsolver
+
+#Rewards (3 states x 2 actions)
+#e.g. choosing second action in first state gives reward=-1
+rewards = [[5,-1],
+           [1,-2],
+           [50,0]]
+
+#Transition probabilities (3 from_states x 2 actions x 3 to_states)
+#e.g. choosing first action in third state gives a probability of 0.6 of staying in third state
+tranMatWithZeros = [[[0.9,0.1,0.0],[0.1,0.9,0.0]],
+                    [[0.4,0.5,0.1],[0.3,0.5,0.2]],
+                    [[0.2,0.2,0.6],[0.5,0.5,0.0]]]
+```
+
+Now, create the model object and insert the problem parameters.
+
+```python
+#Create model object
+mdl = mdpsolver.model()
+
+#Insert the problem parameters
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatWithZeros=tranMatWithZeros)
+```
+
+We can now optimize the policy.
+
+```python
+mdl.solve()
+```
+
+The optimized policy can be returned in a variety of ways. Here, we return the policy as a list and print directly in the terminal. 
+
+```python
+print(mdl.getPolicy())
+#[1, 1, 0]
+```
+
+## Large transition matrix?
+
+`mdpsolver` has three alternative formats for large and highly sparse transition probability matrices.
+
+**(1) Elementwise representation (excluding elements containing zeros):**
+
+```python
+#[from_state,action,to_state,probability]
+tranMatElementwise = [[0,0,0,0.9],
+                      [0,0,1,0.1],
+                      [0,1,0,0.1],
+                      [0,1,1,0.9],
+                      [1,0,0,0.4],
+                      [1,0,1,0.5],
+                      [1,0,2,0.1],
+                      [1,1,0,0.3],
+                      [1,1,1,0.5],
+                      [1,1,2,0.2],
+                      [2,0,0,0.2],
+                      [2,0,1,0.2],
+                      [2,0,2,0.6],
+                      [2,1,0,0.5],
+                      [2,1,1,0.5]]
+
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatElementwise=tranMatElementwise)
+```
+
+**(2) Probabilities and column (to_state) indices in separate lists:**
+
+```python
+tranMatProbs = [[[0.9,0.1],[0.1,0.9]],
+                [[0.4,0.5,0.1],[0.3,0.5,0.2]],
+                [[0.2,0.2,0.6],[0.5,0.5]]]
+
+tranMatColumns = [[[0,1],[0,1]],
+                [[0,1,2],[0,1,2]],
+                [[0,1,2],[0,1]]]
+
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatProbs=tranMatProbs,
+        tranMatColumns=tranMatColumns)
+
+```
+
+**(3) Load the elementwise representation from a file:**
+
+```python
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatFromFile="transitions.csv")
+```
+
+# User manual
+
+Further information can be found in the wiki for `mdpsolver` on Github.
+
+# How to cite
+
+[![DOI](https://zenodo.org/badge/294063917.svg)](https://zenodo.org/badge/latestdoi/294063917)

mdpsolver-0.9.1/README.md

@@ -0,0 +1,131 @@
+# `mdpsolver`: An efficient solver for Markov Decision Processes
+ 
+`mdpsolver` is the Python package for deriving the policies of Markov Decision Processes (MDPs) with infinite-horizon and the expected total discounted reward optimality criterion. 
+
+## Features
+
+* Available on PyPI.
+* Solver-engine developed in C++.
+* Available optimization algorithms: *Value iteration*, *Policy iteration*, and *Modified policy iteration*.  
+* Includes a variety of input formats available for users to choose from.
+
+# Quick start guide
+
+The following shows how to get quickly started with `mdpsolver`.
+
+## Installation
+
+Download and install `mdpsolver` directly from PyPI. 
+
+```
+pip install mdpsolver
+```
+
+## Usage
+
+Start by specifying the reward function and transition probabilities as lists. The following is an example of a simple MDP containing **three states** and **two actions in each state**.
+
+```python
+#Import packages
+import mdpsolver
+
+#Rewards (3 states x 2 actions)
+#e.g. choosing second action in first state gives reward=-1
+rewards = [[5,-1],
+           [1,-2],
+           [50,0]]
+
+#Transition probabilities (3 from_states x 2 actions x 3 to_states)
+#e.g. choosing first action in third state gives a probability of 0.6 of staying in third state
+tranMatWithZeros = [[[0.9,0.1,0.0],[0.1,0.9,0.0]],
+                    [[0.4,0.5,0.1],[0.3,0.5,0.2]],
+                    [[0.2,0.2,0.6],[0.5,0.5,0.0]]]
+```
+
+Now, create the model object and insert the problem parameters.
+
+```python
+#Create model object
+mdl = mdpsolver.model()
+
+#Insert the problem parameters
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatWithZeros=tranMatWithZeros)
+```
+
+We can now optimize the policy.
+
+```python
+mdl.solve()
+```
+
+The optimized policy can be returned in a variety of ways. Here, we return the policy as a list and print directly in the terminal. 
+
+```python
+print(mdl.getPolicy())
+#[1, 1, 0]
+```
+
+## Large transition matrix?
+
+`mdpsolver` has three alternative formats for large and highly sparse transition probability matrices.
+
+**(1) Elementwise representation (excluding elements containing zeros):**
+
+```python
+#[from_state,action,to_state,probability]
+tranMatElementwise = [[0,0,0,0.9],
+                      [0,0,1,0.1],
+                      [0,1,0,0.1],
+                      [0,1,1,0.9],
+                      [1,0,0,0.4],
+                      [1,0,1,0.5],
+                      [1,0,2,0.1],
+                      [1,1,0,0.3],
+                      [1,1,1,0.5],
+                      [1,1,2,0.2],
+                      [2,0,0,0.2],
+                      [2,0,1,0.2],
+                      [2,0,2,0.6],
+                      [2,1,0,0.5],
+                      [2,1,1,0.5]]
+
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatElementwise=tranMatElementwise)
+```
+
+**(2) Probabilities and column (to_state) indices in separate lists:**
+
+```python
+tranMatProbs = [[[0.9,0.1],[0.1,0.9]],
+                [[0.4,0.5,0.1],[0.3,0.5,0.2]],
+                [[0.2,0.2,0.6],[0.5,0.5]]]
+
+tranMatColumns = [[[0,1],[0,1]],
+                [[0,1,2],[0,1,2]],
+                [[0,1,2],[0,1]]]
+
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatProbs=tranMatProbs,
+        tranMatColumns=tranMatColumns)
+
+```
+
+**(3) Load the elementwise representation from a file:**
+
+```python
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatFromFile="transitions.csv")
+```
+
+# User manual
+
+Further information can be found in the wiki for `mdpsolver` on Github.
+
+# How to cite
+
+[![DOI](https://zenodo.org/badge/294063917.svg)](https://zenodo.org/badge/latestdoi/294063917)

mdpsolver-0.9.1/pyproject.toml

@@ -0,0 +1,30 @@
+[build-system]
+requires = ["setuptools>=59.6.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+include-package-data = true
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[project]
+name = "mdpsolver"
+version = "0.9.1"
+authors = [
+  { name="Anders Reenberg Andersen", email="andersra@live.dk" },
+  { name="Jesper Fink Andersen", email="jfan@dtu.dk" },
+]
+description = "An efficient solver for Markov Decision Processes"
+readme = "README.md"
+requires-python = ">=3.10"
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+
+[project.urls]
+Homepage = "https://github.com/areenberg/MDPSolver"
+Issues = "https://github.com/areenberg/MDPSolver/issues"

mdpsolver-0.9.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

mdpsolver-0.9.1/src/mdpsolver/__init__.py

@@ -0,0 +1 @@
+from mdpsolver.model import model

mdpsolver-0.9.1/src/mdpsolver/model.py

@@ -0,0 +1,131 @@
+from mdpsolver import solvermodule
+
+#auxilary methods
+from mdpsolver.problems.tbmaux import tbmaux
+from mdpsolver.problems.cbmaux import cbmaux
+
+class model:
+    
+    def __init__(self):
+        self.initialize()            
+        
+    def initialize(self):
+        #create the solver object
+        self.mdl = solvermodule.Model()
+    
+    def solve(self,
+              algorithm="mpi",
+              tolerance=1e-3,
+              update="standard",
+              parIterLim=100,
+              SORrelaxation=1.0,
+              initPolicy=list(),
+              initValueVector=list(),
+              verbose=False):
+        #solves the problem employing the specified algorithm
+        self.mdl.solve(algorithm=algorithm,
+              tolerance=tolerance,
+              update=update,
+              parIterLim=parIterLim,
+              SORrelaxation=SORrelaxation,
+              initPolicy=initPolicy,
+              initValueVector=initValueVector,
+              verbose=verbose)
+    
+    def getRuntime(self):
+        #returns the runtime in milliseconds
+        return self.mdl.getRuntime()
+        
+    def printPolicy(self):
+        #print the entire policy in the terminal
+        self.mdl.printPolicy()
+    
+    def printValueVector(self):
+        #print the entire value vector in the terminal
+        self.mdl.printValueVector()
+    
+    def getAction(self,stateIndex=0):
+        #returns an action from the optimized policy
+        return self.mdl.getAction(stateIndex=stateIndex)
+    
+    def getValue(self,stateIndex=0):
+        #returns a value from the optimized value vector
+        return self.mdl.getValue(stateIndex=stateIndex)
+    
+    def getPolicy(self):
+        #returns the entire policy
+        return self.mdl.getPolicy()
+    
+    def getValueVector(self):
+        #returns the entire value vector
+        return self.mdl.getValueVector()
+    
+    def saveToFile(self,fileName="result.csv",type="policy"):
+        #saves the optimized policy or value vector to a comma-separated file 
+        return self.mdl.saveToFile(fileName=fileName,type=type)
+    
+    def mdp(self,
+            discount=0.99,
+            rewards=list(),
+            rewardsElementwise=list(),
+            rewardsFromFile="rewards.csv",
+            tranMatWithZeros=list(),
+            tranMatElementwise=list(),
+            tranMatProbs=list(),
+            tranMatColumns=list(),
+            tranMatFromFile="transitions.csv"
+            ):
+        #the generic MDP model
+        self.mdl.mdp(discount=discount,
+            rewards=rewards,
+            rewardsElementwise=rewardsElementwise,
+            rewardsFromFile=rewardsFromFile,
+            tranMatWithZeros=tranMatWithZeros,
+            tranMatElementwise=tranMatElementwise,
+            tranMatProbs=tranMatProbs,
+            tranMatColumns=tranMatColumns,
+            tranMatFromFile=tranMatFromFile)
+
+    def tbm(self,discount=0.99,
+            components=2,
+            stages=10,
+            replacementCost=-10,
+            setupCost=-10,
+            unexpectedFailureCost=-20,
+            expiredNotFixedCost=-1e6,
+            failureProb=0.1,
+            failureProbMin=0.01,
+            failureProbHat=0.1):
+        #the time-based maintenance problem
+        self.mdl.tbm(discount=discount,
+                     components=components,
+                     stages=stages,
+                     replacementCost=-10,
+                     setupCost=setupCost,
+                     unexpectedFailureCost=unexpectedFailureCost,
+                     expiredNotFixedCost=expiredNotFixedCost,
+                     failureProb=failureProb,
+                     failureProbMin=failureProbMin,
+                     failureProbHat=failureProbHat)
+        self.aux = tbmaux(components=components,stages=stages)
+    
+    def cbm(self,discount=0.99,
+            components=2,
+            stages=10,
+            pCompMat=list(),
+            preventiveCost=-5,
+            correctiveCost=-11,
+            setupCost=-4,
+            failurePenalty=-300,
+            kOfN=-1):
+        #the condition-based maintenance problem
+        self.mdl.cbm(discount=discount,
+                     components=components,
+                     stages=stages,
+                     pCompMat=pCompMat,
+                     preventiveCost=preventiveCost,
+                     correctiveCost=correctiveCost,
+                     setupCost=setupCost,
+                     failurePenalty=failurePenalty,
+                     kOfN=kOfN)
+        self.aux = cbmaux(components=components,stages=stages)

mdpsolver-0.9.1/src/mdpsolver/problems/cbmaux.py

@@ -0,0 +1,39 @@
+import math
+
+#Condition-based maintenance problem
+
+#Note: The class is only used for retrieving/converting information
+#(i.e. returning an action of the policy). The policy is optimized 
+#in the 'model' (model.py) class.
+
+class cbmaux:
+
+    def __init__(self,components,stages):
+        self.components=components
+        self.stages=stages
+        
+    def compToState(self,compStates):
+        #input: list containing the stage of each component
+        #output: the associated state index
+        prod = self.components * [1]
+        if self.components>1:
+            for i in range(self.components-2,-1,-1):
+                prod[i]=prod[i+1]*self.stages 
+        stateIndex=0
+        for i in range(self.components):
+            stateIndex += compStates[i]*prod[i]    
+        return stateIndex
+    
+    def actionToReplace(self,actionIndex):
+        #input: action index from the optimized policy
+        #output: list of bools indicating which comps to replace
+        prod = self.components * [1]
+        if self.components>1:
+            for i in range(self.components-2,-1,-1):
+                prod[i]=prod[i+1]*2
+        replace = self.components * [False]        
+        for i in range(self.components):
+            if  math.floor(actionIndex/prod[i])>0:
+                replace[i]=True
+                actionIndex-=prod[i]             
+        return replace

mdpsolver-0.9.1/src/mdpsolver/problems/tbmaux.py

@@ -0,0 +1,39 @@
+import math
+
+#Time-based maintenance problem
+
+#Note: The class is only used for retrieving/converting information
+#(i.e. returning an action of the policy). The policy is optimized 
+#in the 'model' (model.py) class.
+
+class tbmaux:
+
+    def __init__(self,components,stages):
+        self.components=components
+        self.stages=stages
+        
+    def compToState(self,compStates):
+        #input: list containing the stage of each component
+        #output: the associated state index
+        prod = self.components * [1]
+        if self.components>1:
+            for i in range(self.components-2,-1,-1):
+                prod[i]=prod[i+1]*self.stages 
+        stateIndex=0
+        for i in range(self.components):
+            stateIndex += compStates[i]*prod[i]    
+        return stateIndex
+    
+    def actionToReplace(self,actionIndex):
+        #input: action index from the optimized policy
+        #output: list of bools indicating which comps to replace
+        prod = self.components * [1]
+        if self.components>1:
+            for i in range(self.components-2,-1,-1):
+                prod[i]=prod[i+1]*2
+        replace = self.components * [False]        
+        for i in range(self.components):
+            if  math.floor(actionIndex/prod[i])>0:
+                replace[i]=True
+                actionIndex-=prod[i]             
+        return replace

mdpsolver-0.9.1/src/mdpsolver.egg-info/PKG-INFO

@@ -0,0 +1,146 @@
+Metadata-Version: 2.1
+Name: mdpsolver
+Version: 0.9.1
+Summary: An efficient solver for Markov Decision Processes
+Author-email: Anders Reenberg Andersen <andersra@live.dk>, Jesper Fink Andersen <jfan@dtu.dk>
+Project-URL: Homepage, https://github.com/areenberg/MDPSolver
+Project-URL: Issues, https://github.com/areenberg/MDPSolver/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# `mdpsolver`: An efficient solver for Markov Decision Processes
+ 
+`mdpsolver` is the Python package for deriving the policies of Markov Decision Processes (MDPs) with infinite-horizon and the expected total discounted reward optimality criterion. 
+
+## Features
+
+* Available on PyPI.
+* Solver-engine developed in C++.
+* Available optimization algorithms: *Value iteration*, *Policy iteration*, and *Modified policy iteration*.  
+* Includes a variety of input formats available for users to choose from.
+
+# Quick start guide
+
+The following shows how to get quickly started with `mdpsolver`.
+
+## Installation
+
+Download and install `mdpsolver` directly from PyPI. 
+
+```
+pip install mdpsolver
+```
+
+## Usage
+
+Start by specifying the reward function and transition probabilities as lists. The following is an example of a simple MDP containing **three states** and **two actions in each state**.
+
+```python
+#Import packages
+import mdpsolver
+
+#Rewards (3 states x 2 actions)
+#e.g. choosing second action in first state gives reward=-1
+rewards = [[5,-1],
+           [1,-2],
+           [50,0]]
+
+#Transition probabilities (3 from_states x 2 actions x 3 to_states)
+#e.g. choosing first action in third state gives a probability of 0.6 of staying in third state
+tranMatWithZeros = [[[0.9,0.1,0.0],[0.1,0.9,0.0]],
+                    [[0.4,0.5,0.1],[0.3,0.5,0.2]],
+                    [[0.2,0.2,0.6],[0.5,0.5,0.0]]]
+```
+
+Now, create the model object and insert the problem parameters.
+
+```python
+#Create model object
+mdl = mdpsolver.model()
+
+#Insert the problem parameters
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatWithZeros=tranMatWithZeros)
+```
+
+We can now optimize the policy.
+
+```python
+mdl.solve()
+```
+
+The optimized policy can be returned in a variety of ways. Here, we return the policy as a list and print directly in the terminal. 
+
+```python
+print(mdl.getPolicy())
+#[1, 1, 0]
+```
+
+## Large transition matrix?
+
+`mdpsolver` has three alternative formats for large and highly sparse transition probability matrices.
+
+**(1) Elementwise representation (excluding elements containing zeros):**
+
+```python
+#[from_state,action,to_state,probability]
+tranMatElementwise = [[0,0,0,0.9],
+                      [0,0,1,0.1],
+                      [0,1,0,0.1],
+                      [0,1,1,0.9],
+                      [1,0,0,0.4],
+                      [1,0,1,0.5],
+                      [1,0,2,0.1],
+                      [1,1,0,0.3],
+                      [1,1,1,0.5],
+                      [1,1,2,0.2],
+                      [2,0,0,0.2],
+                      [2,0,1,0.2],
+                      [2,0,2,0.6],
+                      [2,1,0,0.5],
+                      [2,1,1,0.5]]
+
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatElementwise=tranMatElementwise)
+```
+
+**(2) Probabilities and column (to_state) indices in separate lists:**
+
+```python
+tranMatProbs = [[[0.9,0.1],[0.1,0.9]],
+                [[0.4,0.5,0.1],[0.3,0.5,0.2]],
+                [[0.2,0.2,0.6],[0.5,0.5]]]
+
+tranMatColumns = [[[0,1],[0,1]],
+                [[0,1,2],[0,1,2]],
+                [[0,1,2],[0,1]]]
+
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatProbs=tranMatProbs,
+        tranMatColumns=tranMatColumns)
+
+```
+
+**(3) Load the elementwise representation from a file:**
+
+```python
+mdl.mdp(discount=0.8,
+        rewards=rewards,
+        tranMatFromFile="transitions.csv")
+```
+
+# User manual
+
+Further information can be found in the wiki for `mdpsolver` on Github.
+
+# How to cite
+
+[![DOI](https://zenodo.org/badge/294063917.svg)](https://zenodo.org/badge/latestdoi/294063917)

mdpsolver-0.9.1/src/mdpsolver.egg-info/SOURCES.txt

@@ -0,0 +1,14 @@
+LICENSE
+MANIFEST.in
+README.md
+pyproject.toml
+src/mdpsolver/__init__.py
+src/mdpsolver/model.py
+src/mdpsolver/solvermodule.cp310-win_amd64.pyd
+src/mdpsolver/solvermodule.cpython-310-x86_64-linux-gnu.so
+src/mdpsolver.egg-info/PKG-INFO
+src/mdpsolver.egg-info/SOURCES.txt
+src/mdpsolver.egg-info/dependency_links.txt
+src/mdpsolver.egg-info/top_level.txt
+src/mdpsolver/problems/cbmaux.py
+src/mdpsolver/problems/tbmaux.py

mdpsolver-0.9.1/src/mdpsolver.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

mdpsolver-0.9.1/src/mdpsolver.egg-info/top_level.txt

@@ -0,0 +1 @@
+mdpsolver