neuralnetworknumpy 0.2.0__tar.gz → 0.3.0__tar.gz

neuralnetworknumpy-0.3.0/PKG-INFO

@@ -0,0 +1,394 @@
+Metadata-Version: 2.4
+Name: neuralnetworknumpy
+Version: 0.3.0
+Summary: A neural network framework built completely from scratch using NumPy
+Author: Itamar Senderovitz
+License: MIT
+Project-URL: Homepage, https://github.com/Sendy45/NeuralNetworkFromScratch
+Keywords: neural network,deep learning,machine learning,numpy
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.21
+Requires-Dist: tqdm>=4.60
+
+# neuralnetworknumpy
+
+A deep learning framework built from scratch using NumPy. Implements forward propagation, backpropagation, convolutional layers, recurrent layers, transformer blocks, residual connections, and common optimizers — no PyTorch or TensorFlow required.
+
+---
+
+## Installation
+
+```bash
+pip install neuralnetworknumpy
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/Sendy45/NeuralNetworkFromScratch.git
+cd neuralnetworknumpy
+pip install .
+```
+
+**Dependencies:** `numpy`, `tqdm`  
+**Optional (for examples):** `keras` (datasets only)
+
+---
+
+## Quick Start
+
+### Dense network (MNIST)
+
+```python
+import numpy as np
+from keras.datasets import mnist
+from neuralnetworknumpy import NeuralNetwork
+from neuralnetworknumpy.layers import Dense, ReLu, BatchNorm, Dropout, Softmax
+
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+X_train = X_train.reshape(-1, 784).astype(np.float32) / 255.0
+X_test  = X_test.reshape(-1, 784).astype(np.float32)  / 255.0
+
+model = NeuralNetwork([
+    Dense(256), ReLu(), BatchNorm(), Dropout(0.2),
+    Dense(128), ReLu(), BatchNorm(),
+    Dense(10),  Softmax()
+])
+
+model.compile(optimizer="adam", loss_type="cross_entropy", lr=0.001)
+history = model.fit(X_train, y_train, X_val=X_test, y_val=y_test, epochs=10, batch_size=64)
+print(f"Val accuracy: {model.evaluate(X_test, y_test):.4f}")
+```
+
+### Convolutional network with residual blocks (Fashion-MNIST)
+
+```python
+import numpy as np
+from keras.datasets import fashion_mnist
+from neuralnetworknumpy import NeuralNetwork
+from neuralnetworknumpy.layers import (
+    Conv2D, DepthwiseSeparableConv2D, MaxPooling2D, AveragePooling2D,
+    Flatten, Dense, ReLu, Softmax, BatchNorm2D, ResidualBlock
+)
+
+(train_X, train_y), (test_X, test_y) = fashion_mnist.load_data()
+train_X = train_X.astype(np.float32) / 255.0
+test_X  = test_X.astype(np.float32)  / 255.0
+train_X = train_X[..., np.newaxis]   # (N, 28, 28, 1)
+test_X  = test_X[..., np.newaxis]
+
+def res_block(in_ch, out_ch, stride=(1, 1)):
+    projection = None
+    if in_ch != out_ch or stride != (1, 1):
+        projection = Conv2D(out_ch, 1, strides=stride, padding="same")
+    return ResidualBlock([
+        Conv2D(out_ch, (3, 3), strides=stride, padding="same"),
+        BatchNorm2D(), ReLu(),
+        DepthwiseSeparableConv2D(out_ch, (3, 3), padding="same"),
+        BatchNorm2D(), ReLu(),
+    ], projection=projection)
+
+model = NeuralNetwork([
+    Conv2D(16, (3, 3), padding="same"), BatchNorm2D(), ReLu(),
+    MaxPooling2D((2, 2)),               # → 14×14×16
+    res_block(16, 32),
+    MaxPooling2D((2, 2)),               # → 7×7×32
+    res_block(32, 64),
+    AveragePooling2D((7, 7)),           # global avg pool → 1×1×64
+    Flatten(),
+    Dense(128), ReLu(),
+    Dense(10),  Softmax()
+])
+
+model.compile(optimizer="adam", loss_type="cross_entropy", lr=0.001, lambda_=0.0001)
+history = model.fit(train_X, train_y, X_val=test_X, y_val=test_y, epochs=20, batch_size=64)
+model.save("fashion_model")
+```
+
+### Transformer language model (WikiText)
+
+```python
+import numpy as np
+from neuralnetworknumpy import NeuralNetwork
+from neuralnetworknumpy.layers import (
+    Embedding, PositionEmbedding, TransformerBlock, Dense, Softmax
+)
+from neuralnetworknumpy.tokenizer import Tokenizer
+from neuralnetworknumpy.learning_rate import LinearWarmup, CosineDecay, SequentialLR
+
+tokenizer = Tokenizer()
+tokenizer.load("tokenizer.json")
+vocab_size = len(tokenizer.vocab)
+
+SEQ_LEN   = 64
+EMBED_DIM = 256
+N_HEADS   = 8
+N_BLOCKS  = 4
+D_FFN     = EMBED_DIM * 4
+
+model = NeuralNetwork([
+    Embedding(vocab_size, EMBED_DIM),
+    PositionEmbedding(SEQ_LEN),
+    *[TransformerBlock(EMBED_DIM, N_HEADS, D_FFN) for _ in range(N_BLOCKS)],
+    Dense(vocab_size, EMBED_DIM),
+    Softmax()
+])
+
+total_steps  = 40 * (500_000 // 64)
+warmup_steps = 4000
+
+schedule = SequentialLR(
+    schedules=[
+        LinearWarmup(warmup_steps=warmup_steps, max_lr=0.001),
+        CosineDecay(max_steps=total_steps - warmup_steps, base_lr=0.001),
+    ],
+    boundaries=[warmup_steps]
+)
+
+model.compile(optimizer="adam", loss_type="cross_entropy", lr=schedule, task="language_model")
+model.fit(X_train, y_train, X_val=X_val, y_val=y_val, epochs=40, batch_size=64)
+
+# Generate text
+print(model.generate("the researchers discovered", tokenizer, max_new_tokens=50, seq_len=SEQ_LEN))
+```
+
+### Seq2Seq with LSTM
+
+```python
+from neuralnetworknumpy import NeuralNetwork
+from neuralnetworknumpy.layers import Seq2Seq
+from neuralnetworknumpy.tokenizer import Tokenizer
+
+tokenizer = Tokenizer()
+tokenizer.load("tokenizer.json")
+
+model = NeuralNetwork([
+    Seq2Seq(vocab_size=len(tokenizer.vocab), embed_dim=128, hidden_size=256, layer_type="LSTM")
+])
+
+model.compile(optimizer="adam", loss_type="cross_entropy", lr=0.001, task="language_model")
+model.fit((X_src, X_trg), y, epochs=20, batch_size=64)
+
+print(model.generate("how are you", tokenizer, max_new_tokens=30, mode="seq2seq"))
+```
+
+---
+
+## API Reference
+
+### `NeuralNetwork`
+
+```python
+model = NeuralNetwork(layers)
+```
+
+| Method | Description |
+|---|---|
+| `compile(loss_type, optimizer, lr, lambda_, task)` | Set training hyperparameters |
+| `fit(X, y, X_val, y_val, epochs, batch_size)` | Train the model, returns `History` |
+| `predict(X)` | Returns class label predictions |
+| `predict_proba(X)` | Returns raw output activations |
+| `evaluate(X, y)` | Returns accuracy |
+| `save(path)` | Serialise model to `.pkl` |
+| `NeuralNetwork.load(path)` | Load a saved model |
+| `summary()` | Print layer descriptions and parameter counts |
+| `generate(prompt_ids, tokenizer, ...)` | Autoregressive text generation |
+| `check_gradient(X, y)` | Numerical gradient check for debugging |
+
+**Optimizers:** `"adam"`, `"adamW"`, `"momentum"`, `"rmsprop"`, `"sgd"`  
+**Loss functions:** `"cross_entropy"`, `"mse"`  
+**Tasks:** `"classification"`, `"language_model"`
+
+---
+
+### Layers
+
+#### Dense
+
+| Layer | Constructor | Notes |
+|---|---|---|
+| `Dense` | `Dense(units)` | Fully connected |
+| `BatchNorm` | `BatchNorm(momentum=0.9)` | For 1D feature vectors |
+| `Dropout` | `Dropout(rate)` | Dropped during training only |
+
+#### Activations
+
+`ReLu()` · `Sigmoid()` · `Softmax()` · `Tanh()` · `Linear()`
+
+#### Convolutional
+
+All conv layers expect input shape `(batch, H, W, channels)`.
+
+| Layer | Constructor | Notes |
+|---|---|---|
+| `Conv2D` | `Conv2D(filters, kernel_size, strides, padding)` | Standard 2D convolution |
+| `GroupConv2D` | `GroupConv2D(filters, kernel_size, groups, strides, padding)` | `groups=1` → Conv2D, `groups=C_in` → depthwise |
+| `DepthwiseConv2D` | `DepthwiseConv2D(kernel_size, strides, padding)` | One filter per input channel |
+| `DepthwiseSeparableConv2D` | `DepthwiseSeparableConv2D(filters, kernel_size, strides, padding)` | Depthwise + pointwise |
+| `SpatiallySeparableConv2D` | `SpatiallySeparableConv2D(filters, kernel_size)` | Row × column factored convolution |
+| `BatchNorm2D` | `BatchNorm2D(momentum=0.9)` | Normalises over spatial + batch dims |
+
+#### Pooling
+
+| Layer | Constructor | Notes |
+|---|---|---|
+| `MaxPooling2D` | `MaxPooling2D(pool_size, strides, padding)` | Takes max in each window |
+| `AveragePooling2D` | `AveragePooling2D(pool_size, strides, padding)` | Takes average in each window |
+| `GlobalAveragePooling2D` | `GlobalAveragePooling2D()` | Collapses H×W → 1 per channel |
+
+#### Structural
+
+| Layer | Constructor | Notes |
+|---|---|---|
+| `Flatten` | `Flatten()` | `(m, H, W, C)` → `(m, H*W*C)` |
+| `ResidualBlock` | `ResidualBlock(layers, projection=None)` | Skip connection |
+
+#### Recurrent
+
+| Layer | Constructor | Notes |
+|---|---|---|
+| `RNN` | `RNN(embed_dim, hidden_size)` | Simple recurrent, suffers from vanishing gradient |
+| `GRU` | `GRU(embed_dim, hidden_size)` | Gated — update + reset gates |
+| `LSTM` | `LSTM(embed_dim, hidden_size)` | Gated — forget, input, output, cell |
+| `Seq2Seq` | `Seq2Seq(vocab_size, embed_dim, hidden_size, layer_type)` | Encoder-decoder with RNN/GRU/LSTM |
+
+#### Transformer
+
+| Layer | Constructor | Notes |
+|---|---|---|
+| `Embedding` | `Embedding(vocab_size, embed_dim)` | Token id → dense vector |
+| `PositionEmbedding` | `PositionEmbedding(seq_len)` | Learnable positional encoding |
+| `TransformerBlock` | `TransformerBlock(model_dim, n_heads, ffn_dim)` | Causal self-attention + FFN + LayerNorm |
+| `MultiHeadAttention` | `MultiHeadAttention(embed_dim, heads_num)` | Multi-head scaled dot-product attention |
+| `LayerNorm` | `LayerNorm(embed_dim)` | Normalises over feature dimension |
+
+---
+
+### Learning Rate Schedules
+
+```python
+from neuralnetworknumpy.learning_rate import (
+    LinearWarmup, CosineDecay, StepDecay, ExponentialDecay, SequentialLR
+)
+```
+
+| Schedule | Constructor | Notes |
+|---|---|---|
+| `LinearWarmup` | `LinearWarmup(warmup_steps, max_lr)` | Ramps from 0 to `max_lr` |
+| `CosineDecay` | `CosineDecay(max_steps, base_lr)` | Cosine annealing to 0 |
+| `StepDecay` | `StepDecay(drop_rate, step_size, base_lr)` | Drops by factor every N steps |
+| `ExponentialDecay` | `ExponentialDecay(drop_rate, base_lr)` | Continuous exponential decay |
+| `SequentialLR` | `SequentialLR(schedules, boundaries)` | Chains schedules together |
+
+**Warmup + cosine (recommended for transformers):**
+
+```python
+schedule = SequentialLR(
+    schedules=[
+        LinearWarmup(warmup_steps=4000, max_lr=0.001),
+        CosineDecay(max_steps=total_steps - 4000, base_lr=0.001),
+    ],
+    boundaries=[4000]
+)
+model.compile(optimizer="adam", lr=schedule, ...)
+```
+
+---
+
+### Masking
+
+```python
+from neuralnetworknumpy.masks import causal_mask, padding_mask, combined_mask
+```
+
+| Function | Description |
+|---|---|
+| `causal_mask(T)` | Upper-triangle mask — blocks future positions |
+| `padding_mask(token_ids, pad_id)` | Blocks PAD tokens |
+| `combined_mask(token_ids, pad_id)` | Causal + padding combined |
+
+---
+
+### Tokenizer
+
+```python
+from neuralnetworknumpy.tokenizer import Tokenizer
+
+tokenizer = Tokenizer()
+tokenizer.fit(text, vocab_size=8000)   # train BPE tokenizer
+tokenizer.save("tokenizer.json")
+tokenizer.load("tokenizer.json")
+
+ids  = tokenizer.encode("hello world")
+text = tokenizer.decode(ids)
+```
+
+**Download pretrained tokenizer (trained on WikiText-103):**
+
+```python
+from neuralnetworknumpy.pretrained import download_tokenizer
+
+path = download_tokenizer()   # downloads tokenizer.json if not present
+tokenizer.load(path)
+```
+
+---
+
+### Text Generation
+
+```python
+# Transformer LM
+output = model.generate(
+    prompt_ids   = tokenizer.encode("the scientists discovered"),
+    tokenizer    = tokenizer,
+    max_new_tokens = 50,
+    temperature  = 0.8,
+    seq_len      = 64,
+    mode         = "transformer",
+    top_k        = 10
+)
+
+# Seq2Seq
+output = model.generate(
+    prompt_ids   = tokenizer.encode("how are you"),
+    tokenizer    = tokenizer,
+    max_new_tokens = 30,
+    mode         = "seq2seq"
+)
+```
+
+---
+
+### Utilities
+
+```python
+from neuralnetworknumpy.utils import History, Scaler, split_train_test, split_train_validation
+```
+
+| Utility | Description |
+|---|---|
+| `Scaler(mode)` | `"standard"` or `"minmax"` normalisation |
+| `split_train_test(X, y, test_ratio)` | Random train/test split |
+| `split_train_validation(X, y, val_ratio)` | Random train/val split |
+| `History` | Returned by `model.fit()` — tracks loss and metrics per epoch |
+
+---
+
+## Save and Load
+
+```python
+model.save("my_model")                    # writes my_model.pkl
+model2 = NeuralNetwork.load("my_model")   # loads it back
+print(model2.evaluate(X_test, y_test))
+```
+
+---
+
+## Project Structure