online-cp 0.0.1__tar.gz

online_cp-0.0.1/LICENCE

@@ -0,0 +1,8 @@
+Copyright 2024 Johan Hallberg Szabadváry
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

online_cp-0.0.1/PKG-INFO

@@ -0,0 +1,108 @@
+Metadata-Version: 2.1
+Name: online-cp
+Version: 0.0.1
+Summary: An implementation of online conformal prediction
+Author-email: Johan Hallberg Szabadvary <johan.hallberg.szabadvary@ju.se>
+Project-URL: Homepage, https://github.com/egonmedhatten/OnlineConformalPrediction
+Project-URL: Bug Tracker, https://github.com/egonmedhatten/OnlineConformalPrediction/issues
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Operating System :: OS Independent
+Description-Content-Type: text/markdown
+License-File: LICENCE
+Requires-Dist: numpy
+Requires-Dist: scipy
+
+# online-cp -- Online Conformal Prediction
+
+This project is an implementation of Online Conformal Prediction.
+
+For now, take a look at [`example.ipynb`](example.ipynb) to see how to use the library.
+
+
+## Quick start
+Let's create a dataset with noisy evaluations of the function $f(x_1, x_2) = x_1 + x_2$.
+
+```py
+import numpy as np
+N = 30
+X = np.random.uniform(0, 1, (N, 2))
+y = X.sum(axis=1) + np.random.normal(0, 0.1, N)
+cp.learn_initial_training_set(X, y)
+```
+
+Import the library and create a regressor:
+
+```py
+from online_cp import ConformalRidgeRegressor
+cp = ConformalRidgeRegressor()
+```
+
+Alternative 1: Learn the whole dataset online
+```py
+cp.learn_initial_training_set(X, y)
+```
+
+Predict an object (your output may not be exactly the same, as the dataset depends on the random seed).
+```py
+cp.predict(np.array([0.5, 0.5]), epsilon=0.1, bounds='both')
+(0.8065748777057368, 1.2222461945130274)
+```
+The prediction set is the closed interval whose boundaries are indicated by the output.
+
+Alternative 2: Learn the dataset sequentially online, and make predictions as we go. In order to output nontrivial prediction at significance level $\epsilon=0.1$, we need to have learned at least 20 examples.
+```py
+cp = ConformalRidgeRegressor()
+for i, (obj, lab) in enumerate(zip(X, y)):
+    print(cp.predict(obj, epsilon=0.1, bounds='both'))
+    cp.learn_one(obj, lab)
+```
+The output will be ```(inf, inf)``` for the first 19 predictions, after which we will typically see meaningful prediction sets.
+
+
+## Future considerations
+
+### Release minimal version 
+For use in projects, it may be good to have a released minimal version of OnlineConformalPrediction. Initially, it could include
+* Conformalised Ridge Regression
+* Plugin martingale
+* Possibly Conformalised Nearest Neighbours Regression (but I will have to check it carefully for any bugs)
+
+### Properties of CPs?
+* Should we keep track of errors internally in the parent class? 
+* Should we store the average interval size?
+* For classifiers; should we store the efficiency metrics?
+
+### Linear regression
+We will initally focus on regression, but online classification is actually easier. A simple class that uses e.g. scikit-learn classifiers to define nonconformity measure could be easily implemented. 
+
+There are at least three commonly used regularisations used in linear regression, all of which are compatible with the kernel trick. 
+* $L1$ (Lasso)
+* $L2$ (Ridge)
+* Linear combination of the above (Elastic net)
+
+All of these can be conformalized, and at least Ridge can also be used in conformal predictive systems (CPS).
+
+Another relatively simple regressor is the k-nearest neighbours algorithm, which is very flexible as it can use arbitrary distances. It is particularly interesting in the CPS setting. The distance can be measured in feature space as defined by a kernel.
+
+Ridge and KNN are described in detail in Algorithmic Learning in a Random World. Lasso and Elastic net are conformalised in the paper Fast Exact Conformalization of Lasso using Piecewise Linear Homotopy, but I am unaware of any extention to CPS. 
+
+### Teaching schedule
+Section 3.3 in Algorithmic Learning in a Radnom World deals with, so called, weak teachers. In the pure online mode, labels arrive immediately after a predition is made. This makes little sense in practice. The notion of a teaching schedule formalises this, and makes the relevant validity guarantees clear. There are three types of validity; weak, strong, and iterated logartihm validity. 
+
+There may be settings where the user wants to specify a teaching schedule beforehand, to guarantee some property of validity. It may also be the case that the teaching schedule is implied by the usage, and it would then be useful to know if the resulting prediciton sets are valid.
+
+A teaching schedule also serves as documentation of what has been done, which could be useful in practice.
+
+## Todo
+* Should we add some scaler? Don't know if it is neccesary for Ridge
+* Possibly add a class MimoConformalRidgeRegressor
+* Add CPS version of ridge regressor?
+* Possibly add a TeachingSchedule?
+* Possibly add ACI, both for single, and MIMO CRR?
+* Add references to papers and books to README
+* Add k-NN regressor and CPS
+
+# References
+How to cite papers? I think I have seen it in some repos.

online_cp-0.0.1/README.md

@@ -0,0 +1,92 @@
+# online-cp -- Online Conformal Prediction
+
+This project is an implementation of Online Conformal Prediction.
+
+For now, take a look at [`example.ipynb`](example.ipynb) to see how to use the library.
+
+
+## Quick start
+Let's create a dataset with noisy evaluations of the function $f(x_1, x_2) = x_1 + x_2$.
+
+```py
+import numpy as np
+N = 30
+X = np.random.uniform(0, 1, (N, 2))
+y = X.sum(axis=1) + np.random.normal(0, 0.1, N)
+cp.learn_initial_training_set(X, y)
+```
+
+Import the library and create a regressor:
+
+```py
+from online_cp import ConformalRidgeRegressor
+cp = ConformalRidgeRegressor()
+```
+
+Alternative 1: Learn the whole dataset online
+```py
+cp.learn_initial_training_set(X, y)
+```
+
+Predict an object (your output may not be exactly the same, as the dataset depends on the random seed).
+```py
+cp.predict(np.array([0.5, 0.5]), epsilon=0.1, bounds='both')
+(0.8065748777057368, 1.2222461945130274)
+```
+The prediction set is the closed interval whose boundaries are indicated by the output.
+
+Alternative 2: Learn the dataset sequentially online, and make predictions as we go. In order to output nontrivial prediction at significance level $\epsilon=0.1$, we need to have learned at least 20 examples.
+```py
+cp = ConformalRidgeRegressor()
+for i, (obj, lab) in enumerate(zip(X, y)):
+    print(cp.predict(obj, epsilon=0.1, bounds='both'))
+    cp.learn_one(obj, lab)
+```
+The output will be ```(inf, inf)``` for the first 19 predictions, after which we will typically see meaningful prediction sets.
+
+
+## Future considerations
+
+### Release minimal version 
+For use in projects, it may be good to have a released minimal version of OnlineConformalPrediction. Initially, it could include
+* Conformalised Ridge Regression
+* Plugin martingale
+* Possibly Conformalised Nearest Neighbours Regression (but I will have to check it carefully for any bugs)
+
+### Properties of CPs?
+* Should we keep track of errors internally in the parent class? 
+* Should we store the average interval size?
+* For classifiers; should we store the efficiency metrics?
+
+### Linear regression
+We will initally focus on regression, but online classification is actually easier. A simple class that uses e.g. scikit-learn classifiers to define nonconformity measure could be easily implemented. 
+
+There are at least three commonly used regularisations used in linear regression, all of which are compatible with the kernel trick. 
+* $L1$ (Lasso)
+* $L2$ (Ridge)
+* Linear combination of the above (Elastic net)
+
+All of these can be conformalized, and at least Ridge can also be used in conformal predictive systems (CPS).
+
+Another relatively simple regressor is the k-nearest neighbours algorithm, which is very flexible as it can use arbitrary distances. It is particularly interesting in the CPS setting. The distance can be measured in feature space as defined by a kernel.
+
+Ridge and KNN are described in detail in Algorithmic Learning in a Random World. Lasso and Elastic net are conformalised in the paper Fast Exact Conformalization of Lasso using Piecewise Linear Homotopy, but I am unaware of any extention to CPS. 
+
+### Teaching schedule
+Section 3.3 in Algorithmic Learning in a Radnom World deals with, so called, weak teachers. In the pure online mode, labels arrive immediately after a predition is made. This makes little sense in practice. The notion of a teaching schedule formalises this, and makes the relevant validity guarantees clear. There are three types of validity; weak, strong, and iterated logartihm validity. 
+
+There may be settings where the user wants to specify a teaching schedule beforehand, to guarantee some property of validity. It may also be the case that the teaching schedule is implied by the usage, and it would then be useful to know if the resulting prediciton sets are valid.
+
+A teaching schedule also serves as documentation of what has been done, which could be useful in practice.
+
+## Todo
+* Should we add some scaler? Don't know if it is neccesary for Ridge
+* Possibly add a class MimoConformalRidgeRegressor
+* Add CPS version of ridge regressor?
+* Possibly add a TeachingSchedule?
+* Possibly add ACI, both for single, and MIMO CRR?
+* Add references to papers and books to README
+* Add k-NN regressor and CPS
+
+# References
+How to cite papers? I think I have seen it in some repos.

online_cp-0.0.1/pyproject.toml

@@ -0,0 +1,28 @@
+[project]
+name = "online-cp"
+version = "0.0.1"
+authors = [
+	{ name="Johan Hallberg Szabadvary", email="johan.hallberg.szabadvary@ju.se" },
+]
+description = "An implementation of online conformal prediction"
+readme = "README.md"
+dependencies = [
+	'numpy',
+	'scipy',
+]
+classifiers = [
+	"Development Status :: 2 - Pre-Alpha",
+	"Programming Language :: Python :: 3",
+	"License :: OSI Approved :: BSD License",
+    "Operating System :: OS Independent",
+]
+
+[project.urls]
+"Homepage"    = "https://github.com/egonmedhatten/OnlineConformalPrediction"
+"Bug Tracker" = "https://github.com/egonmedhatten/OnlineConformalPrediction/issues"
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+
+[tool.setuptools.packages.find]
+where = ["src"]

online_cp-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

online_cp-0.0.1/src/online_cp/CPS.py

@@ -0,0 +1,292 @@
+import numpy as np
+import time
+import warnings
+from scipy.spatial.distance import pdist, cdist, squareform
+
+MACHINE_EPSILON = np.finfo(np.float64).eps
+
+class ConformalPredictiveSystem:
+    '''
+    Parent class for conformal predictive systems. Unclear if some methods are common to all, so perhaps we don't need it.
+    '''
+
+
+class NearestNeighboursPredictionMachine(ConformalPredictiveSystem):
+
+    def __init__(self, k, distance='euclidean', distance_func=None, warnings=True, verbose=0, rnd_state=None):
+        '''
+        Consider adding possibility to update self.k as the training set grows, e.g. by some heuristic or something.
+        Two rules of thumb are quite simple:
+            1. Choose k close to sqrt(n) where n is the training set size
+            2. If the data has large variance, choose k larger. If the variance is small, choose k smaller. This is less clear, however.
+        '''
+        
+        self.k = k
+
+        self.distance = distance
+        if distance_func is None:
+            self.distance_func = self._standard_distance_func
+        else:
+            self.distance_func = distance_func
+            self.distance = 'custom'
+
+        self.X = None
+        self.y = None
+        self.D = None
+
+        # Should we raise warnings
+        self.warnings = warnings
+
+        self.verbose = verbose
+        self.rnd_gen = np.random.default_rng(rnd_state)
+
+    def _standard_distance_func(self, X, y=None):
+        '''
+        By default we use scipy to compute distances
+        '''
+        X = np.atleast_2d(X)
+        if y is None:
+            dists = squareform(pdist(X, metric=self.distance))
+        else:
+            y = np.atleast_2d(y)
+            dists = cdist(X, y, metric=self.distance)
+        return dists
+    
+
+    def learn_initial_training_set(self, X, y):
+        self.X = X
+        self.y = y
+        self.D = self.distance_func(X)
+
+    
+    @staticmethod
+    def update_distance_matrix(D, d):
+        return np.block([[D, d], [d.T, np.array([0])]])
+    
+
+    def learn_one(self, x, y, precomputed=None):
+        '''
+        precomputed is a dictionary
+        {
+            'X': X,
+            'D': D,
+        }
+        '''
+        # Learn label y
+        if self.y is None:
+            self.y = np.array([y])
+        else:
+            self.y = np.append(self.y, y)
+
+        if precomputed is None:
+            # Learn object x
+            if self.X is None:
+                self.X = x.reshape(1,-1)
+                self.D = self.distance_func(self.X)
+            else:
+                d = self.distance_func(self.X, x)
+                self.D = self.update_distance_matrix(self.D, d)
+                self.X = np.append(self.X, x.reshape(1, -1), axis=0)
+        else:
+            self.X = precomputed['X']
+            self.D = precomputed['D']
+
+    
+    def predict_cpd(self, x, return_update=False, save_time=False):
+
+        '''
+        TODO Add possibility to return precomputed as we did with ConformalRegressor.
+        '''
+        tic = time.time()
+        # Temporarily update the distance matrix
+        if self.X is None:
+            X = x.reshape(1,-1)
+            D = self.distance_func(X)
+            y = np.array(-np.inf) # Initialise label as -inf
+        else:
+            d = self.distance_func(self.X, x)
+            D = self.update_distance_matrix(self.D, d)
+            X = np.append(self.X, x.reshape(1, -1), axis=0)
+            y = np.append(self.y, -np.inf) # Initialise label as -inf
+        toc_dist = time.time()-tic
+
+        tic = time.time()
+        # Find all neighbours and semi-neighbours
+        # NOTE I have a feeling that this could be stored in order to save time... Investigate later
+        k_nearest = D.argsort(axis=0)[1:self.k+1]
+        toc_sort = time.time() - tic
+
+        tic = time.time()
+        idx_all_neighbours_and_semi_neighbours = []
+
+        full_neighbours = []
+        single_neighbours = []
+        semi_neighbours = []
+
+        n = D.shape[0] - 1
+        k_nearest_of_n = k_nearest.T[n]
+        for i, col in enumerate(k_nearest.T):
+            if i in k_nearest_of_n and n in col:
+                # print(f'{i} is a full neighbour')
+                idx_all_neighbours_and_semi_neighbours.append(i)
+                full_neighbours.append(y[i])
+            if i in k_nearest_of_n and n not in col:
+                # print(f'{i} is a single neighbour')
+                idx_all_neighbours_and_semi_neighbours.append(i)
+                single_neighbours.append(y[i])
+            if i not in k_nearest_of_n and n in col:
+                # print(f'{i} is a semi-neighbour')
+                idx_all_neighbours_and_semi_neighbours.append(i)
+                semi_neighbours.append(y[i])
+        toc_find_neighbours = time.time() - tic
+        
+        # Line 1
+        Kprime = len(idx_all_neighbours_and_semi_neighbours)
+        # Line 2 and 3
+        Y = np.zeros(shape=(Kprime+2,))
+        Y[0] = -np.inf
+        Y[-1] = np.inf
+        Y[1:-1] = y[idx_all_neighbours_and_semi_neighbours]
+        Y.sort()
+
+        # Line 4
+        Alpha = -np.inf * np.ones(n + 1) # Initialize at something unreasonable
+        N = -np.inf * np.ones(self.k + 1) # Initialize at something unreasonable
+        for i in range(n + 1):
+            J = k_nearest.T[i]
+            Alpha[i] = np.where(y[J] <= y[i])[0].shape[0]
+        for k in range(self.k + 1):
+            N[k] = np.where(Alpha == k)[0].shape[0]
+        
+        # Line 5
+        L = -np.inf * np.ones(Kprime+1) # Initialize at something unreasonable
+        U = -np.inf * np.ones(Kprime+1) # Initialize at something unreasonable
+        L[0] = 0
+        U[0] = N[0]/(n+1)
+
+        tic = time.time()
+        # Line 6
+        for k in range(1, Kprime + 1):
+            # FIXME Something is wrong with this loop... Very difficult to tell what.
+            # Line 7
+            
+            if Y[k] in full_neighbours or Y[k] in single_neighbours:
+                # Line 8
+                N[int(Alpha[-1])] -= 1
+                Alpha[n] += 1
+                N[int(Alpha[-1])] += 1
+
+            # Line 9
+            if Y[k] in full_neighbours or Y[k] in semi_neighbours:
+                # Line 10
+                N[int(Alpha[k])] -= 1
+                Alpha[k] -= 1
+                N[int(Alpha[k])] += 1
+            
+            # Line 11
+            L[k] = N[:int(Alpha[-1])].sum() / (n + 1) if Alpha[-1] != 0  else 0
+            U[k] = N[:int(Alpha[-1]) + 1].sum() / (n + 1)
+        toc_loop = time.time() - tic
+
+        time_dict = {
+            'Compute distance matrix': toc_dist,
+            'Sort distance matrix': toc_sort,
+            'Find all neighbours and semi-neighbours': toc_find_neighbours,
+            'Loop': toc_loop
+        }
+        time_dict = time_dict if save_time else None
+        # Line 12
+        cps = KnnConformalPredictiveDistributionFunction(L, U, Y, time_dict)
+
+        if return_update:
+            return cps, {'X': X, 'D': D}
+        else:
+            return cps
+    
+
+class ConformalPredictiveDistributionFunction:
+
+    '''
+    NOTE
+    The CPD contains all the information needed to form a
+    prediction set. We can take quantiles and so on.
+    '''
+
+    def quantile(self, quantile, tau):
+        raise NotImplementedError('Parent class has not quantile function')
+
+
+    def predict_set(self, tau, epsilon=0.1, bounds='both'):
+        '''
+        The convex hull of the epsilon/2 and 1-epsilon/2 quantiles make up
+        the prediction set Gamma(epsilon)
+        TODO Add things like bounds, so we can predict just upper or lower...
+             Make it possible to pass two epsilons to predict skewed intervals...
+        '''
+        q1 = epsilon/2
+        q2 = 1 - epsilon/2
+        if bounds=='both':
+            lower = self.quantile(q1, tau)
+            upper = self.quantile(q2, tau)
+        elif bounds=='lower':
+            lower = self.quantile(q1, tau)
+            upper = np.inf
+        elif bounds=='upper':
+            lower = -np.inf
+            upper = self.quantile(q2, tau)
+        else: 
+            raise Exception
+
+        # print(f'Lower: {lower}')
+        # print(f'Upper: {upper}')
+        return lower, upper
+    
+
+    # These methods relate to when the cpd is used to predict sets
+    @staticmethod
+    def err(Gamma, y):
+        return int(not(Gamma[0] <= y <= Gamma[1]))
+    
+
+    @staticmethod
+    def width(Gamma):
+        return Gamma[1] - Gamma[0]
+
+
+
+class KnnConformalPredictiveDistributionFunction(ConformalPredictiveDistributionFunction):
+
+    def __init__(self, L, U, Y, time_dict=None):
+        self.L = L 
+        self.U = U
+        self.Y = Y
+
+        self.time_dict = time_dict
+
+    def __call__(self, y, tau=None):
+        Y = self.Y[:-1]
+        idx_eq = np.where(y == Y)[0]
+        if idx_eq.shape[0] > 0:
+            k = idx_eq.min()
+            interval = (self.L[k-1], self.U[k])
+        else:
+            k = np.where(Y <= y)[0].max()
+            interval = (self.L[k], self.U[k])
+        
+        Pi0 = interval[0]
+        Pi1 = interval[1]
+
+        if tau is None:
+            return Pi0, Pi1
+        else:
+            return (1 - tau) * Pi0 + tau * Pi1
+        
+    
+    def quantile(self, quantile, tau):
+        q = np.inf
+        for y in self.Y[::-1]:
+            if self.__call__(y, tau) >= quantile:
+                q = y
+            else:
+                return q
+        return q

online_cp-0.0.1/src/online_cp/__init__.py

@@ -0,0 +1,5 @@
+#!/usr/bin/env python
+
+from .regressors import ConformalRidgeRegressor, ConformalNearestNeighboursRegressor
+from .classifiers import ConformalNearestNeighboursClassifier 
+from .martingale import PluginMartingale