mlpclassifier 0.1.0__tar.gz

mlpclassifier-0.1.0/PKG-INFO

@@ -0,0 +1,82 @@
+Metadata-Version: 2.4
+Name: mlpclassifier
+Version: 0.1.0
+Summary: An implementation of an MLP classifier (with an interface of that of scikit-learn's MLPClassifier class.
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: matplotlib>=3.10.8
+Requires-Dist: numpy>=2.4.3
+Requires-Dist: pandas>=3.0.1
+Requires-Dist: scikit-learn>=1.8.0
+
+# Neural-Network-From-Scratch-COSC-221-CSB
+
+Neural Network to classify handwritten digits (a rite of passage project at this point lol).
+
+We will try to re-implement a stripped down version of the `MLPClassifier` class from `scikit-learn` from first principles. With this, we can then train a general classifier using the Multi-Layered Perceptron model.
+
+# To run
+
+So since we've re-implemented an MLP using `scikit-learn`'s `MLPClassifier` as a template, the API should be familiar.
+
+To import
+
+
+# Dataset
+
+Download the dataset from Kaggle
+
+```
+curl -L https://www.kaggle.com/api/v1/datasets/download/hojjatk/mnist-dataset -o ./dataset.zip
+```
+
+Then just unzip it into a directory called `./dataset`
+
+```
+unzip -d dataset ./dataset.zip
+```
+
+Optional, but clean redundancy:
+
+```
+rm -r *-idx*-ubyte
+```
+
+I've removed some duplicates, so currently I have:
+
+```
+$ ls ./dataset/
+ t10k-images.idx3-ubyte   train-images.idx3-ubyte
+ t10k-labels.idx1-ubyte   train-labels.idx1-ubyte
+```
+
+So it seems like by convention:
+
+- we divide our dataset into training data, and then testing data
+- currently, it seems like we have 60k training examples and 10k testing examples
+- they do this to see how well the model has generalized
+
+# Reference model
+
+For now, we'll use a reference model through `scikit-learn`. 
+
+# TODO
+
+- [x] debug all the row vector stuff
+- [] package it in pip
+- [] document the API
+
+## Forward propagation
+
+- [x] variable L for layer
+- [x] a list $n^{[l]}$ for the size at each layer
+- [] initialize using He's initalization
+- [x] forward propagation step using that forward propagation formula
+
+## Backward propagation
+
+- [x] He's initialization
+- [x] back propagation
+- [x] scoring
+- [x] saving
+- [] make the learn rate $\alpha$ more adjustable

mlpclassifier-0.1.0/README.md

@@ -0,0 +1,71 @@
+# Neural-Network-From-Scratch-COSC-221-CSB
+
+Neural Network to classify handwritten digits (a rite of passage project at this point lol).
+
+We will try to re-implement a stripped down version of the `MLPClassifier` class from `scikit-learn` from first principles. With this, we can then train a general classifier using the Multi-Layered Perceptron model.
+
+# To run
+
+So since we've re-implemented an MLP using `scikit-learn`'s `MLPClassifier` as a template, the API should be familiar.
+
+To import
+
+
+# Dataset
+
+Download the dataset from Kaggle
+
+```
+curl -L https://www.kaggle.com/api/v1/datasets/download/hojjatk/mnist-dataset -o ./dataset.zip
+```
+
+Then just unzip it into a directory called `./dataset`
+
+```
+unzip -d dataset ./dataset.zip
+```
+
+Optional, but clean redundancy:
+
+```
+rm -r *-idx*-ubyte
+```
+
+I've removed some duplicates, so currently I have:
+
+```
+$ ls ./dataset/
+ t10k-images.idx3-ubyte   train-images.idx3-ubyte
+ t10k-labels.idx1-ubyte   train-labels.idx1-ubyte
+```
+
+So it seems like by convention:
+
+- we divide our dataset into training data, and then testing data
+- currently, it seems like we have 60k training examples and 10k testing examples
+- they do this to see how well the model has generalized
+
+# Reference model
+
+For now, we'll use a reference model through `scikit-learn`. 
+
+# TODO
+
+- [x] debug all the row vector stuff
+- [] package it in pip
+- [] document the API
+
+## Forward propagation
+
+- [x] variable L for layer
+- [x] a list $n^{[l]}$ for the size at each layer
+- [] initialize using He's initalization
+- [x] forward propagation step using that forward propagation formula
+
+## Backward propagation
+
+- [x] He's initialization
+- [x] back propagation
+- [x] scoring
+- [x] saving
+- [] make the learn rate $\alpha$ more adjustable

mlpclassifier-0.1.0/pyproject.toml

@@ -0,0 +1,12 @@
+[project]
+name = "mlpclassifier"
+version = "0.1.0"
+description = "An implementation of an MLP classifier (with an interface of that of scikit-learn's MLPClassifier class."
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "matplotlib>=3.10.8",
+    "numpy>=2.4.3",
+    "pandas>=3.0.1",
+    "scikit-learn>=1.8.0",
+]

mlpclassifier-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

mlpclassifier-0.1.0/src/classifier/classifier.py

@@ -0,0 +1,271 @@
+import pickle
+
+from src.utils import *
+import copy
+
+class MLPClassifier:
+
+    activations = {
+        "relu":(relu,d_relu),
+        "logistic":(sigmoid, d_sigmoid),
+        "tanh":(tanh, d_tanh),
+        "identity":(identity, d_identity)
+    }
+
+    def __init__(self, hidden_layer_sizes:tuple[int,...], max_iter:int=20, activation:str='relu', batch_size:int=200, alpha:float=1e-2, verbose:bool=False) -> None:
+        """
+        We basically are implementing a stripped down version of the scikit learn MLPClassifier class. No solvers, we're only going to use stochastic gradient descent.
+
+        Args:
+            hidden_layer_sizes (tuple): a tuple representing the size of each hidden layer
+            max_iter (int): the number of passes through the whole dataset
+            activation (string): the activation function, can be relu, sigmoid, tanh, identity
+            alpha (float): the learning rate. Default is 1e-10.
+            verbose (bool): a boolean representing whether we want to display progress to the user
+
+        Return:
+            None
+        """
+        if hidden_layer_sizes == tuple():
+            raise ValueError("The hidden layer must exist")
+        self.hidden_layer_sizes = hidden_layer_sizes
+        self.max_iter = max_iter
+        self.verbose = verbose
+        self.alpha = alpha
+        self.batch_size = batch_size
+
+        if activation not in self.activations:
+            raise ValueError(f"The activation function called {activation} doesn't exist")
+
+        self.activation, self.d_activation = self.activations[activation]
+
+    def load_weights(self, coefs, intercepts, classes):
+        """
+        Load the weights and biases that has already been trained
+        """
+
+        # weights contains a list of weight matrix
+        expected_weights_shape = tuple((self.hidden_layer_sizes[i],self.hidden_layer_sizes[i + 1])
+            for i in range(len(self.hidden_layer_sizes) - 1))
+        # (256, 128), (128, 64), (64, 10)
+        actual_weights_shape = tuple(w.shape for w in coefs[1:-1])
+        if actual_weights_shape != expected_weights_shape:
+            raise ValueError(f"Please load weights of the right shape for this model. Expected hidden layer weight shape {expected_weights_shape}")
+
+        expected_biases_shape = tuple((b, )
+            for b in self.hidden_layer_sizes)
+        actual_biases_shape = tuple(b.shape for b in intercepts[:-1])
+        if actual_biases_shape != expected_biases_shape:
+            raise ValueError(f"Please load biases of the right shape for this model. Expected hidden layer weight shape {expected_biases_shape}, got {actual_biases_shape}"
+)
+        if (coefs[-1].shape[1] != intercepts[-1].shape[0]):
+            raise ValueError("The weights you passed doesn't match that shape of the biases.")
+
+        if (intercepts[-1].shape[0] != len(classes)):
+            raise ValueError("The classes you picked doesn't the size of the biases")
+
+        self.coefs_ = copy.deepcopy(coefs)
+        self.intercepts_ = copy.deepcopy(intercepts)
+        self.classes_ = copy.deepcopy(classes)
+
+    def forward(self, X):
+        """
+        Given a list of samples, we do forward propagation according to the weights and biases to arrive at a prediction.
+
+        Args:
+            X (numpy darray): a numpy array with shape (number of samples, number of features)
+
+        Return:
+            Y (list): a list that contains matrices representing a^{[l]} at layer l, for all m samples.
+            Z (list): a list that contains matrices representing z^{[l]} at layer l, for all m samples.
+        """
+
+        try:
+            # forward propagation
+            Z = [X]
+            Y = [X]
+            for weight, bias in zip(self.coefs_[:-1], self.intercepts_[:-1]):
+                z = Y[-1]@weight + bias
+                Y.append(self.activation(z))
+                Z.append(z)
+            weight = self.coefs_[-1]
+            bias = self.intercepts_[-1]
+            z = Y[-1]@weight + bias
+            Y.append(softmax(z))
+            Z.append(z)
+            return Y, Z
+
+        except NameError as e:
+            raise NameError(f"You can't do forward propagation before acquiring the weights: {e}")
+
+
+    def predict_proba(self, X):
+        """
+        Given a list of samples, we do forward propagation according to the weights and biases to arrive at a prediction.
+
+        Args:
+            X (numpy darray): a numpy array with shape (number of samples, number of features)
+
+        Return:
+            Y (numpy darray): a numpy array with shape (number of sample, number of types of labels) which represents a probability distribution
+        """
+        return self.forward(X)[0][-1]
+
+    def predict(self, X):
+        """
+        A simple wrapper around predict_proba where it automatically maps the probability distribution to the best match among the classes.
+        """
+        Y = self.predict_proba(X)
+        print(Y.shape)
+        return np.argmax(Y, axis=1)
+
+    def score(self, X, Y):
+        """
+        Just gives the percentage at which we predicted the right label.
+
+        Args:
+            X (numpy darray): a list of features
+            Y (numpy darray): a list of labels for those features
+
+        Return:
+            result (tuple): (score, another tuple of index of failed prediction)
+        """
+        boolean_array_result = np.array(self.predict(X)) == np.array(Y)
+        incorrect_indicies = np.where(~boolean_array_result)[0]
+        score = sum(boolean_array_result)/len(boolean_array_result)
+        return score, incorrect_indicies
+
+    def fit(self, X, y, save_path=None) -> None:
+        """
+        This is to train the model, in order to adjust the weights and biases.
+
+        Args:
+            X (numpy darray): a numpy array with shape (number of samples, number of features) 
+            y (numpy darray): the labels with shape (number of sampels, )
+            save_path (string): path to save (optional)
+        Return:
+            None
+        """
+        try:
+            if (len(X) != len(y)):
+                raise ValueError("The number of example features should match that of the number of labels given")
+            # first divide into batches
+            # i hate naming conventions
+            Y = y
+            self.classes_ = np.unique(Y)
+            self.coefs_ = []
+            self.intercepts_= []
+            for i in range(len(self.hidden_layer_sizes) + 1):
+                n = int()
+                n_prev = int()
+                if i == len(self.hidden_layer_sizes):
+                    n = len(self.classes_)
+                else:
+                    n = self.hidden_layer_sizes[i]
+
+                if i == 0:
+                    n_prev = X.shape[1]
+                else:
+                    n_prev = self.hidden_layer_sizes[i - 1]
+                # He's initialization sets variance = 2/n^{[l - 1]}
+                self.coefs_.append(np.random.normal(0, np.sqrt(2/n_prev), size=(n_prev, n)))
+                self.intercepts_.append(np.zeros(n))
+
+            L = len(self.hidden_layer_sizes) + 1
+            for iter_num in range(self.max_iter):
+                J = np.array([])
+                for i in range(int(np.ceil(len(X)/self.batch_size))):
+                    m = self.batch_size
+                    x = X[self.batch_size*i:min(self.batch_size*(i + 1), len(X))]
+                    y = Y[self.batch_size*i:min(self.batch_size*(i + 1), len(Y))]
+                    # one hot labeling
+                    indices= np.searchsorted(self.classes_, y)
+                    y = np.eye(len(self.classes_))[indices]
+
+                    # so we now need to propagate backwards
+
+                    # first need to compute delta[L]
+                    # forward propagation to get a^{[L]}
+                    a, z = self.forward(x)
+
+                    # J for the batch loss matrix
+                    J = cross_entropy_loss(y, a[-1])
+                    # first calculate the batch change in bias
+                    delta = a[-1] - y
+                    for l in reversed(range(1, L)): # l goes from L - 1 to 0
+                        # make db the average already
+                        db = np.mean(delta, axis=0)
+                        dW = (a[l].T @ delta)/m
+
+                        self.coefs_[l] -= self.alpha*dW
+                        self.intercepts_[l] -= self.alpha*db
+
+                        if l > 0:
+                            delta = (delta @ self.coefs_[l].T) * self.d_activation(z[l])
+                if self.verbose:
+                    print(f"Finished iteration {iter_num + 1}/{self.max_iter}. Loss: {np.mean(J, axis=0)}")
+
+        except KeyboardInterrupt as e:
+            if save_path != None:
+                self.save(save_path)
+            raise InterruptedError(f"Training canceled! Has saved weights to {save_path} as specified")
+
+
+    def save(self, path):
+        """
+        Saving weights as a pickle file. Path is just the file path to save at.
+        """
+        with open(path, 'wb') as file:
+                pickle.dump(
+                    {
+                        "weights":self.coefs_,
+                        "biases":self.intercepts_,
+                        "classes":self.classes_
+                    },
+                    file
+                )
+
+
+if __name__ == "__main__":
+    images = get_images_fast("dataset/train-images.idx3-ubyte")
+    labels = get_labels_fast("dataset/train-labels.idx1-ubyte")
+    import matplotlib.pyplot as plt
+    test_images = get_images_fast("dataset/t10k-images.idx3-ubyte")
+    test_labels = get_labels_fast("dataset/t10k-labels.idx1-ubyte")
+
+    X = images.reshape(images.shape[0], -1)/255
+
+    X_test = test_images.reshape(test_images.shape[0], -1)/255
+
+    model = MLPClassifier(
+        hidden_layer_sizes=(128, 64),
+        activation='relu',
+        max_iter=1000,
+        batch_size=200,
+        verbose=True,
+    )
+
+    SAVE_PATH = "weights/self_trained_2.pkl"
+
+    model.fit(X, labels, save_path=SAVE_PATH)
+
+    #with open("weights/sklearn_weights_and_biases.pkl", 'rb') as file:
+    #    weights_and_biases = pickle.load(file)
+    #    model.load_weights(weights_and_biases["weights"],
+    #                       weights_and_biases["biases"],
+    #                       weights_and_biases["classes"])
+
+    N = 10_000
+    score, incorrect_indicies = model.score(X_test[:N], test_labels[:N])
+    print("score: ", score)
+    model.save(SAVE_PATH)
+    #print("incorrect indicies: ", incorrect_indicies)
+    #predict, Z = model.forward(X_test[:N])
+    #print([i.shape for i in Z])
+    #predicted_label = model.predict(np.array([x_test[i] for i in incorrect_indicies]))
+    #print(predicted_label)
+    #for i in incorrect_indicies:
+    #    predicted_label = model.predict(np.array([x_test[i]]))[0]
+    #    plt.imshow(test_images[i])
+    #    plt.title(f"actual label: {test_labels[i]}, predicted label: {predicted_label}")
+    #    plt.show()

mlpclassifier-0.1.0/src/classifier/utils.py

@@ -0,0 +1,110 @@
+import numpy as np
+
+# functions for importing datasets
+
+def get_images(filename):
+    with open(filename, 'rb') as file:
+        magic_bytes = file.read(4)
+        number_of_images = int.from_bytes(file.read(4))
+        number_of_rows = int.from_bytes(file.read(4))
+        number_of_columns = int.from_bytes(file.read(4))
+
+        return [[[int.from_bytes(file.read(1)) for _ in range(number_of_columns)]
+            for _ in range(number_of_rows)]
+            for _ in range(number_of_images)]
+
+def get_labels(filename):
+    with open(filename, 'rb') as file:
+        magic_bytes = file.read(4)
+        number_of_labels = int.from_bytes(file.read(4))
+        return [int.from_bytes(file.read(1)) for _ in range(number_of_labels)]
+
+def get_images_fast(filename):
+    with open(filename, 'rb') as file:
+        magic_bytes = int.from_bytes(file.read(4))
+        if magic_bytes != 2051:
+            raise ValueError("Magic bytes doesn't match that of idx3")
+        number_of_images = int.from_bytes(file.read(4))
+        number_of_rows = int.from_bytes(file.read(4))
+        number_of_columns = int.from_bytes(file.read(4))
+
+        images = np.frombuffer(file.read(), dtype=np.uint8)
+        images = images[:number_of_images*number_of_rows*number_of_columns]
+        return images.reshape(number_of_images, number_of_rows, number_of_columns)
+
+def get_labels_fast(filename):
+    with open(filename, 'rb') as file:
+        magic_bytes = int.from_bytes(file.read(4))
+        if magic_bytes != 2049:
+            raise ValueError("Magic bytes doesn't match that of idx1")
+        number_of_labels = int.from_bytes(file.read(4))
+        return np.frombuffer(file.read(), dtype=np.uint8)
+
+# some useful functions
+
+def relu(z):
+    """
+    The ReLU function makes all input less than 0, 0. Leaves the rest unchanged.
+    """
+    return np.maximum(0, z)
+
+def d_relu(z):
+    """
+    The derivative of the ReLU function is the unit step function.
+    """
+    return z > 0
+
+def sigmoid(z):
+    """
+    The sigmoid function
+    """
+    return 1/(1 + np.exp(-z))
+
+def d_sigmoid(z):
+    """
+    The derivative of the sigmoid function
+    """
+    return sigmoid(z)*(1 - sigmoid(z))
+
+def tanh(z):
+    """
+    The hyperbolic tangent function
+    """
+    # kinda redundant lol
+    return np.tanh(z)
+
+def d_tanh(z):
+    return 1 - tanh(z)**2
+
+def identity(z):
+    return z
+
+def d_identity(z):
+    return 1
+
+def softmax(z):
+    """
+    We're expecting batched z, meaning that we will have m samples as columns, each row being the predicted classification.
+    """
+    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))  # stable softmax
+    #exp_z = np.exp(z)  # stable softmax
+    return exp_z / np.sum(exp_z, axis=1, keepdims=True)
+
+def cross_entropy_loss(y, y_hat):
+    epsilon = 1e-12
+    y_hat = np.clip(y_hat, epsilon, 1 - epsilon) # in case that y_hat is so small that we run out of precision and it becomes 0
+    return - np.sum(y*np.log(y_hat), axis=1)
+
+if __name__ == "__main__":
+    #import matplotlib.pyplot as plt
+    #x = np.linspace(-5, 5, 100)
+    #plt.plot(x, tanh(x), label="g(x)")
+    #plt.legend()
+    #plt.show()
+    #s = softmax(np.array([1,2,3]))
+    #print(s)
+    #print(np.sum(s))
+    pass
+
+
+

mlpclassifier-0.1.0/src/mlpclassifier.egg-info/PKG-INFO

@@ -0,0 +1,82 @@
+Metadata-Version: 2.4
+Name: mlpclassifier
+Version: 0.1.0
+Summary: An implementation of an MLP classifier (with an interface of that of scikit-learn's MLPClassifier class.
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: matplotlib>=3.10.8
+Requires-Dist: numpy>=2.4.3
+Requires-Dist: pandas>=3.0.1
+Requires-Dist: scikit-learn>=1.8.0
+
+# Neural-Network-From-Scratch-COSC-221-CSB
+
+Neural Network to classify handwritten digits (a rite of passage project at this point lol).
+
+We will try to re-implement a stripped down version of the `MLPClassifier` class from `scikit-learn` from first principles. With this, we can then train a general classifier using the Multi-Layered Perceptron model.
+
+# To run
+
+So since we've re-implemented an MLP using `scikit-learn`'s `MLPClassifier` as a template, the API should be familiar.
+
+To import
+
+
+# Dataset
+
+Download the dataset from Kaggle
+
+```
+curl -L https://www.kaggle.com/api/v1/datasets/download/hojjatk/mnist-dataset -o ./dataset.zip
+```
+
+Then just unzip it into a directory called `./dataset`
+
+```
+unzip -d dataset ./dataset.zip
+```
+
+Optional, but clean redundancy:
+
+```
+rm -r *-idx*-ubyte
+```
+
+I've removed some duplicates, so currently I have:
+
+```
+$ ls ./dataset/
+ t10k-images.idx3-ubyte   train-images.idx3-ubyte
+ t10k-labels.idx1-ubyte   train-labels.idx1-ubyte
+```
+
+So it seems like by convention:
+
+- we divide our dataset into training data, and then testing data
+- currently, it seems like we have 60k training examples and 10k testing examples
+- they do this to see how well the model has generalized
+
+# Reference model
+
+For now, we'll use a reference model through `scikit-learn`. 
+
+# TODO
+
+- [x] debug all the row vector stuff
+- [] package it in pip
+- [] document the API
+
+## Forward propagation
+
+- [x] variable L for layer
+- [x] a list $n^{[l]}$ for the size at each layer
+- [] initialize using He's initalization
+- [x] forward propagation step using that forward propagation formula
+
+## Backward propagation
+
+- [x] He's initialization
+- [x] back propagation
+- [x] scoring
+- [x] saving
+- [] make the learn rate $\alpha$ more adjustable

mlpclassifier-0.1.0/src/mlpclassifier.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+README.md
+pyproject.toml
+src/classifier/__init__.py
+src/classifier/classifier.py
+src/classifier/utils.py
+src/mlpclassifier.egg-info/PKG-INFO
+src/mlpclassifier.egg-info/SOURCES.txt
+src/mlpclassifier.egg-info/dependency_links.txt
+src/mlpclassifier.egg-info/requires.txt
+src/mlpclassifier.egg-info/top_level.txt

mlpclassifier-0.1.0/src/mlpclassifier.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

mlpclassifier-0.1.0/src/mlpclassifier.egg-info/requires.txt

@@ -0,0 +1,4 @@
+matplotlib>=3.10.8
+numpy>=2.4.3
+pandas>=3.0.1
+scikit-learn>=1.8.0

mlpclassifier-0.1.0/src/mlpclassifier.egg-info/top_level.txt

@@ -0,0 +1 @@
+classifier