sw-mlp 1.0.0__tar.gz

sw_mlp-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 swangarch and contributer
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sw_mlp-1.0.0/PKG-INFO

@@ -0,0 +1,439 @@
+Metadata-Version: 2.4
+Name: sw-mlp
+Version: 1.0.0
+Summary: A multilayer perceptron implemented with Numpy.
+License-Expression: MIT
+License-File: LICENSE
+Requires-Dist: matplotlib>=3.11.2
+Requires-Dist: numpy>=2.5.4
+Requires-Dist: pandas==2.2.3
+Requires-Python: >=3.12, <3.13
+Description-Content-Type: text/markdown
+
+# Multilayer Perceptron
+
+Multilayer perceptron (MLP) is a simple neural network (NN), can solve both regression and classification tasks. The project provides a reusable and modular implementation.
+
+[[Demo Video]](https://www.youtube.com/watch?v=YRcl1f--tcw)
+
+## Requirements
+
+The project is built from scratch with Python, with only Numpy, Pandas, Matplotlib,
+without high-level ML libraries.
+
+Use Python **3.12** and **uv** to manage the environment and dependencies.
+Install uv before continuing. Run the following commands from the project root:
+
+```bash
+git clone https://github.com/swangarch/multilayer_perceptron
+cd multilayer_perceptron
+
+# Create .venv and install the locked dependencies and the project
+uv sync --locked
+
+# Alternatively, use the Makefile (runs uv sync)
+make setup
+```
+
+Commands below use `uv run`, so activating the virtual environment is not required.
+To activate it manually on macOS/Linux, use `source .venv/bin/activate`.
+
+
+## Usage
+
+Run these commands from the project root. Makefile targets use `uv run -m mlp`.
+
+```bash
+uv run -m mlp --help
+```
+
+### For example dataset classification
+
+```bash
+# To split the dataset
+make split
+
+# To train with training configuration file and training dataset
+make train
+
+# To test with test dataset
+make test
+```
+
+Breast cancer diagnosis dataset, a binary classification based on 30 features of a cell.
+
+### MNIST handwrite digits classification (0–9)
+
+```bash
+# To split, train and test the dataset
+make digit
+```
+
+For the handwritten digit classification task (0–9), the model achieved 97.19% test accuracy using 33,000 training samples of 28 × 28 images in the optimal configuration.
+
+For the handwritten character recognition task with 94 categories, the model reached 91% accuracy based on 175,000 of 28 × 28 images training samples.
+
+### For random generated 1d data regression
+
+```bash
+# To train and test generated data, with relu activation
+make regre-relu
+
+# To train and test generated data, with sigmoid activation
+make regre-sigmoid
+```
+
+## Packaging
+
+Build the wheel and source distribution with uv:
+
+```bash
+uv build
+
+# Alternatively
+make build
+```
+
+The output files are written to `dist/`:
+
+- `sw_mlp-1.0.0-py3-none-any.whl`
+- `sw_mlp-1.0.0.tar.gz`
+
+To install the wheel into a Python 3.12 virtual environment:
+
+```bash
+uv venv --python 3.12
+uv pip install dist/sw_mlp-1.0.0-py3-none-any.whl
+
+# macOS/Linux: run the installed package
+.venv/bin/python -m mlp --help
+```
+
+The wheel contains the Python package. Example configurations and datasets in the
+repository's `config/` and `data/` directories must be distributed separately.
+Pass their paths when running the installed package. The split command writes to
+`data/train.csv` and `data/test.csv`, so the working directory must contain a
+`data/` directory.
+
+## Features
+
+- Training methods → supports **Stochastic Gradient Descent (SGD)** and **mini-batch training**.
+
+- **Visualization** → includes real-time animations to track the learning process.
+
+- **Configurable** architecture → adjustable network shape, activation functions, initialization methods, max iterations, learning rate, batch size, etc via a configuration file.
+
+- Data handling → data loading and preprocessing.
+
+- Support multiple activation functions: **ReLU, Sigmoid, Softmax, Leaky ReLU**
+
+- Support multiple loss functions: 
+  - **Categorical Cross Entropy (CCE)** with automatic one-hot encoding for categorical labels
+  - **Binary Cross Entropy (BCE)**
+  - **Mean Squared Error (MSE)**
+
+- Support **Fine-tuning**: continue training from saved weights for further optimization
+  
+## Methodology
+
+![Demo](visualization/diagram/mlp.jpg)
+
+A **Multilayer Perceptron** is a basic artificial neural network that learns to map input data to the correct output by adjusting internal parameters called weights and biases. It is composed of multiple layers of artificial neurons, where each neuron performs a simple computation — combining inputs, multiplying weights, adding a bias, and passing the result through an activation function to introduce non-linearity. Then using back propagation and gradient descent to find ideal weights and biases.
+
+### Learning Process
+The learning process consists of three main stages:
+
+- **Feedforward** — compute predictions by passing data through all layers.
+- **Back propagation** — calculate gradients of the loss with respect to the weights, using chain rule.
+- **Gradient Descent** — update weights and biases to minimize the loss over time.
+
+Through **repeated iterations** of these steps, the MLP gradually improves its predictions and learns to generalize from the training data.
+
+### Feedfoward
+![Demo](visualization/diagram/feedforward.jpg)
+
+In the feedforward step, the input data passes through several layers of neurons. Each neuron performs a weighted sum of its inputs, adds a bias, and then applies an activation function such as ReLU, Sigmoid, or Softmax.
+
+When neurons are grouped into a layer, their computations can be efficiently expressed using **matrix multiplication**. This allows the model to process many neurons at once, using vectorized operations in **NumPy**. Each layer's output becomes the input to the next layer, forming a feedforward chain that passes information through the network.
+
+$$Z=WX+b$$
+
+$$A=f(Z)$$
+
+The final layer produces the output — either a value (for regression) or class probabilities (for classification).
+
+### Back probagation
+![Demo](visualization/diagram/back_probagation.jpg)
+
+In backpropagation, the model calculates how far its predictions are from the correct answers using a loss function (like MSE or Cross-Entropy). Then, it computes the gradient of this loss to each weight and bias using the **chain rule**. This process tells the model which direction each weight and bias should move to reduce the overall error.
+
+For the output layer, the gradient simplification is elegant:
+
+**MSE Regression**: The output layer typically uses no activation function (linear output). The gradient of the loss with respect to the output is:
+
+$$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$$
+
+**Cross-Entropy Classification**: The output layer uses Softmax (multi-class) or Sigmoid (binary) activation. Remarkably, when combined with Cross-Entropy loss, the gradient simplifies to the same form:
+
+$$\frac{\partial L}{\partial z} = \hat{y} - y$$
+
+
+
+If you're interested in the details, this [**article**](https://shivammehta25.github.io/posts/deriving-categorical-cross-entropy-and-softmax/) by Shivam Mehta explains in depth how this simplification was derived. This mathematical convenience makes backpropagation efficient for both regression and classification tasks.
+
+
+
+### Gradient descent
+![Demo](visualization/diagram/gradient_descent.jpg)
+
+Gradient descent is the optimization process that updates the weights and biases. It moves the weights and biases in the direction that reduces the loss the most, step by step.
+
+$$
+W := W - 𝜂 \frac{\partial J}{\partial W}
+$$
+
+$$
+b := b - 𝜂 \frac{\partial J}{\partial b}
+$$
+
+The learning rate controls how big each step is — too high can make learning unstable, too low can make it too slow.
+
+There are two main modes implemented in this project:
+
+**Stochastic Gradient Descent (SGD)**: updates weights for each training sample, when batch size is set to 1.
+
+**Mini-batch Gradient Descent**: updates weights after a small batch of samples, balancing speed and stability.
+
+### Non linear classification
+
+![Demo](visualization/diagram/classification.jpg)
+
+This diagram shows how an MLP can separate data that is not linearly separable — for instance, when data points form circular or spiral patterns.
+With multiple layers and non-linear activations, the network learns complex boundaries that a simple linear model cannot.
+
+## Programs 
+
+In addition to the provided Makefile, the following section explains the usage in detail and describes how you can customize the training process to fit your specific needs.
+
+### Configuration
+
+
+An json config file to pass network structure and training configuration is required.
+
+```
+      {
+          "shape": [784, 64, 32, 10],  
+          // Network shape, number of neurons in each layer
+
+          "activation_funcs": ["relu", "relu", "softmax"],  
+          // Activation functions for each layer
+
+          "weights_init": ["he", "he", "xavier"],  
+          // Layer weights initialization methods, e.g. he / xavier / zero
+
+          "loss": "CategoricalCrossEntropy",  
+          // Loss function, supporting MeanSquareError and CrossEntropy
+
+          "max_epoch": 1000,  
+          // Maximum number of training epochs
+
+          "learning_rate": 0.01,  
+          // Learning rate for gradient descent
+
+          "batch_size": 50,  
+          // Batch size for mini-batch training， 1 for SGD
+
+          "classification": true,  
+          // Whether to calculate accuracy (true for classification tasks)
+
+          "animation": "none",  
+          // To show training animation, recommended only for 1D data
+
+          "train_ratio": 0.8,  
+          // Ratio to split dataset into training and validation
+
+          "threshold": false,  
+          // Threshold for early stopping, false means disabled
+
+          "index": false  
+          // Whether the input CSV file contains an index column
+      }
+```
+### Train and predict
+We recommand you to use --help option before the first usage.
+```
+      Usage:
+
+        uv run -m mlp <option> <config.json> <data.csv> [params.json]
+
+      Options:
+
+        -s:  Split dataset into (train.csv) and (test.csv), no params required
+
+        -t:  Train dataset, if params.json is provided, it performs fine tuning.
+
+        -p:  Test on a test dataset, if params.json is provided, it performs prediction based on trained model.
+
+        --help:  Show help messages.
+
+        -More features to come.
+
+      Note:
+
+        There is built in dataset generated by script, use --gen-data1d to replace dataset
+```
+
+The program will:
+
+- Defines the network structure (layers, activation functions).  
+
+- Trains using backpropagation and gradient descent.  
+
+- Visualize loss and prediction:
+
+- By default, regression task will visualize the loss, and the data distribution for first dimension.
+
+- In addition, classification task will add an accuracy plot.
+
+
+### Demo
+The example graph on both classification and regression.
+
+![Demo](visualization/nn/demo.png)
+
+
+### Datasets
+
+Datasets are provided in this repository, one example corresponding to breast cancer diagnosis classification.
+It contains 32 columns, where the diagnosis column is the label to predict. The label can take two values:
+
+M → Malignant (cancerous)
+
+B → Benign (non-cancerous)
+
+The remaining 30 numerical features describe various characteristics of a cell nucleus extracted with fine-needle aspiration (FNA), such as radius, texture, perimeter, area, and smoothness. These measurements are widely used for early breast cancer detection.
+
+### To split data:
+
+It is split into two parts: one for training and validation one for testing.
+
+```bash
+uv run -m mlp -s config/data.json data/data.csv
+```
+
+The objective is to classify whether a tumor is malignant or benign based on the given features.
+
+### To train
+```bash
+# Train a new model
+uv run -m mlp -t config/data.json data/train.csv
+
+# Fine-tune an existing model
+uv run -m mlp -t config/data.json data/train.csv params.json
+```
+After training, the params.json will be saved, which contain weights, biases and network structure.
+If params.json is provided, the program will perform fine tuning.
+
+### To test
+```bash
+uv run -m mlp -p config/data.json data/test.csv params.json
+```
+
+The loss function is selected explicitly by the `loss` field in the configuration.
+Use `BinaryCrossEntropy` with a sigmoid output for binary classification,
+`CategoricalCrossEntropy` with a softmax output for multi-class classification,
+or `MeanSquareError` with a linear output (`"none"`) for regression.
+The output activation alone does not select the loss function.
+
+The program will make prediction, an accuracy will be calculated and prediction will be saved in predictions.json.
+
+### Other datasets and configurations provided
+- 0-9.csv _____ 0-9.json => 0 to 9 hand wirte digits recognition with softmax and CCE loss
+- --gen-data1d _____ regre-relu.json => 1d random generated data with RELU activation and MSE loss
+- --gen-data1d _____ regre-sigmoid.json => 1d random generated data with Sigmoid activation and MSE loss
+
+### Class 
+
+Below is a simple example of using the `MultiLayerPerceptron` class to train a regression model.
+
+```python
+from mlp.neural_network import MultiLayerPerceptron, relu
+from mlp.data_process import generate_data_rand
+
+def main():
+    try:
+        # Define network structure: input → 64 → 32 → output
+        net_shape = (1, 64, 32, 1)
+        activation_funcs = (relu, relu, None)  # last layer is linear (regression)
+        weights_init = ("he", "he", "he")  # Define the weights initialization methods
+
+        # Initialize neural network
+        nn = MultiLayerPerceptron(net_shape, activation_funcs, weights_init)
+
+        # Generate training data
+        inputs, truths = generate_data_rand(142, 500, 0.02)
+        test_inputs, test_truths = generate_data_rand(123, 50, 0.02)
+        nn.train(inputs, truths, 10000, 0.005, batch_size=20, animation="plot")
+
+        # Test with new data
+        nn.test(test_inputs, test_truths)
+
+        # Save graphs
+        nn.save_plots()
+
+    except Exception as e:
+        print("Error:", e)
+
+
+if __name__ == "__main__":
+    main()
+```
+
+### Generalization
+
+The figure shows that multilayer perceptron is capable of learning non-linear patterns from different datasets and can generalize beyond the training data, making them applicable to a wide range of tasks.
+
+![Regression demo](visualization/nn/prediction.jpg)
+---
+
+### Hyperparameter Tuning: Batch Size
+
+**Batch size** is a key hyperparameter that influences both training speed and model performance. To study its effect, I kept all other parameters constant and varied only the batch size, trained the model on the MNIST dataset. In this experiment, I used **CuPy** to replace **NumPy** in order to leverage GPU acceleration and improve computational efficiency during training.
+
+```json
+{
+    "shape": [784, 392, 128, 64, 32, 10],
+    "activation_funcs": ["relu", "relu", "relu", "relu", "softmax"],
+    "weights_init": ["he", "he", "he", "he", "xavier"],
+    "loss": "CategoricalCrossEntropy"
+}
+```
+
+| Batch Size | Time (s) | Accuracy | Loss |
+|:-----------:|:--------:|:--------:|:-----:|
+| 1           | 2040     | 0.9814   | 0.1580 |
+| 10          | 180      | 0.9769   | 0.1197 |
+| 50          | 45       | 0.9719   | 0.1371 |
+| 100         | 31       | 0.9676   | 0.1153 |
+| 200         | 25       | 0.9593   | 0.1362 |
+| 500         | 22       | 0.9414   | 0.1937 |
+| 1000        | 21       | 0.9414   | 0.1937 |
+
+#### Analysis
+
+- **Training Time**: As batch size increases, training time decreases dramatically, reflecting better computational efficiency.
+
+- **Accuracy**: Accuracy drops noticeably when batch size exceeds 200, indicating that very large batches may reduce generalization capability.
+The drop in accuracy for large batches can sometimes be reduced by slightly increasing the learning rate.
+
+- **Loss**: Smaller batches tend to achieve lower loss values, suggesting better fine-grained learning at the cost of longer training time.
+
+#### Conclusion
+
+In this experiment, batch sizes between 50 and 100 provide the best balance between training efficiency and model performance.
+Smaller batches has higher accuracy but are slower, while larger batches are faster but may sacrifice precision.
+
+---
+
+## License
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.