mini-jstorch 1.4.5 → 1.6.0

package/README.md

@@ -4,36 +4,28 @@ A lightweight JavaScript neural network library for rapid frontend AI experiment
 
 ## Overview
 
-Mini-JSTorch is a high-performance, minimalist JavaScript library for building neural networks. It runs efficiently in both frontend and backend environments, including low-end devices. The library enables quick experimentation and learning in AI without compromising stability, accuracy, or training reliability.
+Mini-JSTorch is a high-performance, minimalist JavaScript library for building neural networks. It runs efficiently in frontend environments, including low-end devices. The library enables quick experimentation and learning in AI without compromising stability, accuracy, or training reliability.
 
-This release, **version 1.4.5**, We make `Matrix Utils` now can be used in others Files.
+This release, **version 1.6.0:** Adds **LION** Optimizer, Adds **ReduceLROnPlateau** Scheduler, enhanced stability, and improved architecture compatibility.
 
 ---
 
-## Feature Highlights
-
-- **Learning Rate Schedulers:** New `StepLR` and `LambdaLR` for dynamic optimizer learning rate adjustment.
-- **Full Conv2D support:** Forward and backward operations for convolutional layers.
-- **Tensor operations:** Broadcasting, reshaping, and reduction utilities.
-- **Advanced Activations:** Includes `LeakyReLU`, `GELU`, `Mish`, `SiLU`, `ELU`, and more.
-- **Optimizers:** `Adam` and `SGD` with gradient updates.
-- **Dropout Layer:** For regularization during training.
-- **BatchNorm2D:** For stable training in convolutional models.
-- **Tensor Manipulation:** Utilities like `flatten`, `stack`, `concat`, `eye`, `reshape`.
-- **Model Save & Load:** Easy persistence and restore of models.
-- **Test/Demo Templates:** The `tests/` folder provides ready-to-run examples for model building and feature usage.
-- **Performance Optimized:** Suitable for both frontend and backend usage.
-- **Backward Compatibility:** Maintained for core layers and activations.
+## New Features Highlights
+
+- **LIONS Optimizer:** State-of-the-art optimizer with superior stability and convergence
+- **ReduceLROnPlateau Scheduler:** Adaptive learning rate scheduling based on loss plateaus
+- **Enhanced Stability:** Gradient clipping, better weight initialization, and NaN prevention
+
 
 ---
 
 ## Core Features
 
-- **Layers:** Linear, Conv2D  
+- **Layers:** Linear, Flatten, Conv2D  
 - **Activations:** ReLU, Sigmoid, Tanh, LeakyReLU, GELU, Mish, SiLU, ELU  
 - **Loss Functions:** MSELoss, CrossEntropyLoss
-- **Optimizers:** Adam, SGD  
-- **Schedulers:** StepLR, LambdaLR
+- **Optimizers:** Adam, SGD, **LION**  
+- **Schedulers:** StepLR, LambdaLR, **ReduceLROnPlateau**
 - **Regularization:** Dropout, BatchNorm2D  
 - **Utilities:** zeros, randomMatrix, softmax, crossEntropy, dot, addMatrices, reshape, stack, flatten, eye, concat  
 - **Model Container:** Sequential (for stacking layers with forward/backward passes)  
@@ -52,7 +44,7 @@ npm install mini-jstorch
 ## Quick Start Example
 
 ```javascript
-import { Sequential, Linear, ReLU, Sigmoid, CrossEntropyLoss, Adam, StepLR } from 'mini-jstorch';
+import { Sequential, Linear, ReLU, Sigmoid, CrossEntropyLoss, Adam, StepLR } from './src/jstorch.js';
 
 // Build model
 const model = new Sequential([
@@ -99,7 +91,7 @@ predTest.forEach((p,i) => {
 ## Save & Load Models
 
 ```javascript
-import { saveModel, loadModel, Sequential } from './src/MainEngine.js';
+import { saveModel, loadModel, Sequential } from '.jstorch.js';
 
 const json = saveModel(model);
 const model2 = new Sequential([...]); // same architecture
@@ -110,21 +102,23 @@ loadModel(model2, json);
 
 ## Demos & Testing
 
-Check the `tests/` directory for ready-to-run demos:
-- **tests/MakeModel.js:** Build and run a simple neural network.
-- **tests/scheduler.js:** Experiment with learning rate schedulers.
+Check the `demo/` directory for ready-to-run demos:
+- **demo/MakeModel.js:** Build and run a simple neural network.
+- **demo/scheduler.js:** Experiment with learning rate schedulers.
+- **demo/fu_fun.js:** Tests All fu_functions (for users). 
 - Add your own scripts for quick prototyping!
 
 ```bash
-node tests/MakeModel.js
-node tests/scheduler.js
+node demo/MakeModel.js
+node demo/scheduler.js
+node demo/fu_fun.js
 ```
 
 ---
 
 ## Intended Use Cases
 
- - Rapid prototyping of neural networks in frontend and backend.
+ - Rapid prototyping of neural networks in frontend.
  - Learning and teaching foundational neural network concepts.
  - Experimentation on low-end devices or mobile browsers.
  - Lightweight AI projects without GPU dependency.
@@ -135,22 +129,4 @@ node tests/scheduler.js
 
 `MIT License`
 
-**Copyright (c) 2025 rizal-editors**
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+**Copyright (c) 2025 rizal-editors**

package/demo/MakeModel.js

@@ -0,0 +1,36 @@
+import { Sequential, Linear, ReLU, MSELoss, Adam, StepLR, Tanh } from '../src/jstorch.js';
+
+const model = new Sequential([
+    new Linear(2, 16),
+    new Tanh(),
+    new Linear(16, 8), 
+    new ReLU(),
+    new Linear(8, 1)
+]);
+
+const X = [[0,0], [0,1], [1,0], [1,1]];
+const y = [[0], [1], [1], [0]];
+
+const criterion = new MSELoss();
+const optimizer = new Adam(model.parameters(), 0.1);
+const scheduler = new StepLR(optimizer, 25, 0.5); // LR * 0.5 every 25 epochs
+
+console.log("Training Progress:");
+for (let epoch = 0; epoch < 1000; epoch++) {
+    const pred = model.forward(X);
+    const loss = criterion.forward(pred, y);
+    const grad = criterion.backward();
+    model.backward(grad);
+    optimizer.step();
+    scheduler.step();
+    
+    if (epoch % 100 === 0) {
+        console.log(`Epoch ${epoch}: Loss = ${loss.toFixed(6)}, LR = ${optimizer.lr.toFixed(6)}`);
+    }
+}
+
+console.log("\nFinal Predictions:");
+const predictions = model.forward(X);
+predictions.forEach((pred, i) => {
+    console.log(`Input: ${X[i]} -> ${pred[0].toFixed(4)} (target: ${y[i][0]})`);
+});

package/demo/fu_fun.js

@@ -0,0 +1,72 @@
+
+import { 
+    fu_tensor, fu_add, fu_mul, fu_matmul, fu_sum, fu_mean, 
+    fu_relu, fu_sigmoid, fu_tanh, fu_softmax, fu_flatten, fu_reshape 
+} from '../src/jstorch.js';
+
+function testAllFuFunctions() {
+    console.log("TESTING ALL FU_FUNCTIONS\n");
+    
+    // Test 1: fu_tensor
+    console.log("1. fu_tensor");
+    const t1 = fu_tensor([[1, 2], [3, 4]]);
+    console.log("", t1.data);
+    
+    // Test 2: fu_add
+    console.log("\n2. fu_add");
+    const a = fu_tensor([[1, 2]]);
+    const b = fu_tensor([[3, 4]]);
+    const c = fu_add(a, b);
+    console.log("", a.data, "+", b.data, "=", c.data);
+    
+    // Test 3: fu_mul
+    console.log("\n3. fu_mul");
+    const d = fu_mul(a, b);
+    console.log("", a.data, "*", b.data, "=", d.data);
+    
+    // Test 4: fu_matmul
+    console.log("\n4. fu_matmul");
+    const e = fu_tensor([[1, 2]]);
+    const f = fu_tensor([[3], [4]]);
+    const g = fu_matmul(e, f);
+    console.log("matmul =", g.data);
+    
+    // Test 5: fu_sum & fu_mean
+    console.log("\n5. fu_sum & fu_mean");
+    const h = fu_tensor([[1, 2], [3, 4]]);
+    const sum = fu_sum(h);
+    const mean = fu_mean(h);
+    console.log("sum =", sum.data, "mean =", mean.data);
+    
+    // Test 6: fu_relu
+    console.log("\n6. fu_relu");
+    const i = fu_tensor([[-1, 0], [1, 2]]);
+    const relu = fu_relu(i);
+    console.log("relu =", relu.data);
+    
+    // Test 7: fu_sigmoid
+    console.log("\n7. fu_sigmoid");
+    const sigmoid = fu_sigmoid(i);
+    console.log("sigmoid =", sigmoid.data);
+    
+    // Test 8: fu_tanh
+    console.log("\n8. fu_tanh");
+    const tanh = fu_tanh(i);
+    console.log("tanh =", tanh.data);
+    
+    // Test 9: fu_softmax
+    console.log("\n9. fu_softmax");
+    const j = fu_tensor([[1, 2, 3]]);
+    const softmax = fu_softmax(j);
+    console.log("softmax =", softmax.data);
+    
+    // Test 10: fu_flatten & fu_reshape
+    console.log("\n10. fu_flatten & fu_reshape");
+    const k = fu_tensor([[1, 2], [3, 4]]);
+    const flat = fu_flatten(k);
+    const reshaped = fu_reshape(flat, 1, 4);
+    console.log("flatten =", flat.data);
+    console.log("reshape =", reshaped.data);
+}
+
+testAllFuFunctions();

package/demo/scheduler.js

@@ -0,0 +1,69 @@
+// Example: Test ALL learning rate schedulers with mini-jstorch optimizers
+
+import { SGD, StepLR, LambdaLR, ReduceLROnPlateau, Tensor } from "../src/jstorch.js";
+
+const param = { param: [[1, 2], [3, 4]], grad: [[0, 0], [0, 0]] };
+const optimizer = new SGD([param], 0.1);
+
+// --- Test StepLR ---
+console.log("Testing StepLR...");
+const stepScheduler = new StepLR(optimizer, 3, 0.5);
+for (let epoch = 1; epoch <= 10; epoch++) {
+    stepScheduler.step();
+    console.log(`Epoch ${epoch}: LR = ${optimizer.lr.toFixed(4)}`);
+}
+
+// --- Test LambdaLR ---
+console.log("\nTesting LambdaLR...");
+optimizer.lr = 0.1; // Reset LR
+const lambdaScheduler = new LambdaLR(optimizer, epoch => 1.0 / (1 + epoch));
+for (let epoch = 1; epoch <= 5; epoch++) {
+    lambdaScheduler.step();
+    console.log(`Epoch ${epoch}: LR = ${optimizer.lr.toFixed(4)}`);
+}
+
+// --- Test ReduceLROnPlateau ---
+console.log("\nTesting ReduceLROnPlateau...");
+optimizer.lr = 0.1; // Reset LR
+const plateauScheduler = new ReduceLROnPlateau(optimizer, {
+    patience: 2,
+    factor: 0.5,
+    min_lr: 0.01,
+    verbose: true
+});
+
+// Simulate training with plateauing loss
+const losses = [0.9, 0.8, 0.7, 0.69, 0.68, 0.68, 0.68, 0.67, 0.67, 0.67];
+console.log("Simulated training with plateauing loss:");
+for (let epoch = 0; epoch < losses.length; epoch++) {
+    plateauScheduler.step(losses[epoch]);
+    console.log(`Epoch ${epoch + 1}: Loss = ${losses[epoch].toFixed(3)}, LR = ${optimizer.lr.toFixed(4)}, Wait = ${plateauScheduler.wait}`);
+}
+
+// --- Test ReduceLROnPlateau with Cooldown ---
+console.log("\nTesting ReduceLROnPlateau with Cooldown...");
+optimizer.lr = 0.1; // Reset LR
+const plateauWithCooldown = new ReduceLROnPlateau(optimizer, {
+    patience: 2,
+    factor: 0.5,
+    min_lr: 0.01,
+    cooldown: 2,
+    verbose: true
+});
+
+// Simulate training with multiple plateaus
+const losses2 = [0.9, 0.9, 0.9, 0.9, 0.8, 0.8, 0.8, 0.8, 0.7, 0.7];
+console.log("Simulated training with cooldown:");
+for (let epoch = 0; epoch < losses2.length; epoch++) {
+    plateauWithCooldown.step(losses2[epoch]);
+    console.log(`Epoch ${epoch + 1}: Loss = ${losses2[epoch].toFixed(3)}, LR = ${optimizer.lr.toFixed(4)}, Wait = ${plateauWithCooldown.wait}, Cooldown = ${plateauWithCooldown.cooldown_counter}`);
+}
+
+// --- Summary ---
+console.log("\nSCHEDULER SUMMARY:");
+console.log(`StepLR: ${stepScheduler.last_epoch} epochs processed`);
+console.log(`LambdaLR: ${lambdaScheduler.last_epoch} epochs processed`);
+console.log(`ReduceLROnPlateau: ${plateauScheduler.num_reductions} LR reductions`);
+console.log(`ReduceLROnPlateau with Cooldown: ${plateauWithCooldown.num_reductions} LR reductions`);
+
+console.log("\nAll schedulers tested successfully!");

package/index.js

@@ -1,2 +1,2 @@
 // package root
-export * from "./src/MainEngine.js";
+export * from "./src/jstorch.js";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mini-jstorch",
-  "version": "1.4.5",
+  "version": "1.6.0",
   "type": "module", 
   "description": "A lightweight JavaScript neural network library for rapid frontend AI experimentation on low-resource devices Inspired by PyTorch.",
   "main": "index.js",