mini-jstorch 1.6.0 → 1.7.0

package/Docs/About.md

@@ -0,0 +1,83 @@
+# Mini-JSTorch — Technical Information
+
+---
+
+## General Information
+
+- **Project Name:** Mini-JSTorch  
+- **Internal Name:** JST (JSTorch)
+
+> Note:  
+> Early versions of Mini-JSTorch do not strictly follow semantic versioning conventions  
+> (e.g. `0.0.1` for patches, `0.1.0` for minor releases, `1.0.0` for major releases).  
+> This inconsistency reflects the early learning and experimental phase of the project.
+
+---
+
+## 1. Engine Architecture Limitations (JSTorch Core)
+
+This section outlines the known structural weaknesses of the JSTorch engine.  
+Although the architecture may appear complex, it is currently sensitive and tightly coupled.
+
+### Identified Limitations
+
+- **High dependency on Utilities**  
+  Every core class depends directly on the Utilities module, which is defined at the top of the `jstorch.js` file. This creates strong coupling across the engine.
+
+- **Limited Tensor dimensionality**  
+  Tensor implementations currently support only two dimensions.  
+  Extending support to higher-dimensional tensors would require significant architectural changes due to the existing complexity.
+
+- **Uneven class complexity**  
+  New or recently modified classes often become significantly more complex than others, leading to inconsistency in maintainability and internal design balance.
+
+---
+
+## 2. Rationale Behind the `fu_` Utilities
+
+This section explains why the `fu_` utilities were introduced despite the existence of internal Utilities.
+
+### Issues with Internal Utilities
+
+- The Utilities defined at the beginning of `jstorch.js` are **internal engine helpers**, not intended for direct user interaction.
+
+- These Utilities are heavily reused across multiple core classes.  
+  Any modification to a utility function may trigger **cascading (domino) errors** throughout the engine due to tight dependencies.
+
+- Some utility functions intentionally diverge from standard or expected formulas.  
+  For example:
+  - Expected formula:  
+    `Param1 - Param4 * Param3`
+  - Internal Utilities implementation:  
+    `Param1 - Param2 * Param3 + Param4`
+
+  This behavior exists because internal Utilities are optimized for class-level computations, not for user-facing correctness or predictability.
+
+### Purpose of `fu_` Utilities
+
+The `fu_` utilities were designed to improve the **user experience** by providing:
+
+- Predictable and correct computational behavior
+- User-friendly and stable helper functions
+- Isolation from internal engine changes
+- Reduced risk of incorrect outputs and dependency-based cascading errors
+
+In short, `fu_` exists to ensure safety, clarity, and consistency for end users of Mini-JSTorch.
+
+---
+
+## 3. SJK (Shortcut JST Keywords) Reference
+
+This section lists commonly used abbreviations and keywords within the Mini-JSTorch ecosystem.
+
+**Format:** `"KEYWORD" : "Full Name / Meaning"`
+
+- `"JST"` : JSTorch  
+- `"fu"` : For User / User-Friendly  
+- `"fun"` : Function  
+- `"Dummy"` : Experimental  
+- `"Exp"` : Restricted experimental entity  
+- `"msg"` : Message, comment, warning, announcement  
+- `"donot"` : Do not / Don't  
+
+---

package/Docs/Structure.md

@@ -0,0 +1,129 @@
+# Project File Structure # 
+
+This document describes the directory and file structure of the **mini-JSTorch** package.
+It provides an overview of how the project is organized and the purpose of each major component.
+
+---
+
+## Repository Overview
+
+```text
+mini-jstorch/
+├── demo/
+│   ├── fu_fun.js
+│   ├── MakeModel.js
+│   └── scheduler.js
+├── Docs/
+│   ├── About.md
+│   └── Structure.md
+├── src/
+│   ├── jstorch.js
+│   └── Dummy/
+│       └── msg/
+├── index.js
+├── package.json
+└── README.md
+```
+
+---
+
+## Directory Descriptions
+
+`/demo`
+
+- Contains demonstration and testing files.
+
+  - Used for unit testing, quick system checks, and example usage
+  - Intended for users who prefer practical examples over reading full API documentation
+  - Allows testing features without writing extensive manual code
+
+`/Docs`
+
+- Contains detailed documentation related to the mini-JSTorch package.
+
+  - Provides deeper explanations of internal design and usage
+  - Intended for contributors and advanced users
+
+`/src`
+
+- Contains the source code of the JSTorch engine.
+
+  - Houses all core logic and internal implementations
+  - Modifications in this directory directly affect engine behavior
+
+`/src/Dummy`
+
+- Experimental and restricted directory.
+
+  - Used for experimental purposes and future development
+  - Files inside this directory may be unstable or incomplete
+  - Not intended for public or production use
+
+`/src/Dummy/msg`
+
+- Contains warning or message files.
+
+  - Indicates that files within the `Dummy` directory are restricted
+  - Serves as a notification mechanism for experimental or future-update-related content
+
+---
+
+## File Descriptions
+
+`/demo/fu_fun.js`
+
+- Purpose: Tests all user-facing (`fu_`) functions
+- Notes: Focuses on friendly and predictable helper utilities
+
+`/demo/MakeModel.js`
+
+- Purpose: Demonstrates creation of a simple model
+- Notes: Uses the `StepLR` scheduler as part of the example workflow
+
+`/demo/scheduler.js`
+
+- Purpose: Tests scheduler-related functionality
+- Notes: Intended to validate learning rate scheduling behavior
+
+`/Docs/About.md`
+
+- Purpose: Contains additional information about the mini-JSTorch package
+- Notes: May include background, design decisions, or non-API-related explanations
+
+`/Docs/Structure.md`
+
+- Purpose: Documents the repository file and folder structure
+- Notes: This file
+
+`/src/jstorch.js`
+
+- Purpose: Core engine implementation
+
+- Notes:
+
+  - Contains all JSTorch engine logic and functions
+  - Central file of the entire package
+  - Changes here have wide-ranging effects
+
+`index.js`
+
+- Purpose: Package entry point
+- Notes: Exposes public APIs and connects internal modules
+
+`package.json`
+
+- Purpose: Project configuration and metadata
+- Notes: Defines dependencies, scripts, and package information
+
+`README.md`
+
+- Purpose: Main documentation entry
+- Notes: Provides overview, installation instructions, and basic usage
+
+**Notes**
+
+- Experimental files may change or be restricted without notice
+- Users are encouraged to rely on public APIs and documented utilities
+- Internal structures are subject to refactoring as the project evolves
+
+---

package/README.md

@@ -4,18 +4,17 @@ A lightweight JavaScript neural network library for rapid frontend AI experiment
 
 ## Overview
 
-Mini-JSTorch is a high-performance, minimalist JavaScript library for building neural networks. It runs efficiently in frontend environments, including low-end devices. The library enables quick experimentation and learning in AI without compromising stability, accuracy, or training reliability.
+Mini-JSTorch is a high-performance, minimalist JavaScript library for building neural networks. It runs efficiently in Frontend environments, including low-end devices. The library enables quick experimentation and learning in AI without compromising stability, accuracy, or training reliability.
 
-This release, **version 1.6.0:** Adds **LION** Optimizer, Adds **ReduceLROnPlateau** Scheduler, enhanced stability, and improved architecture compatibility.
+This release, **version 1.7.0:** We introduce **Softmax Layer**, **Tokenizer**, **AdamW Optimizer**, and enhanced NLP capabilities.
 
 ---
 
 ## New Features Highlights
 
-- **LIONS Optimizer:** State-of-the-art optimizer with superior stability and convergence
-- **ReduceLROnPlateau Scheduler:** Adaptive learning rate scheduling based on loss plateaus
-- **Enhanced Stability:** Gradient clipping, better weight initialization, and NaN prevention
-
+- **Softmax Layer:** Professional classification output with proper gradient computation
+- **Tokenizer:** Lightweight text preprocessing for NLP tasks
+- **AdamW Optimizer:** Modern optimizer with decoupled weight decay
 
 ---
 
@@ -24,12 +23,17 @@ This release, **version 1.6.0:** Adds **LION** Optimizer, Adds **ReduceLROnPlate
 - **Layers:** Linear, Flatten, Conv2D  
 - **Activations:** ReLU, Sigmoid, Tanh, LeakyReLU, GELU, Mish, SiLU, ELU  
 - **Loss Functions:** MSELoss, CrossEntropyLoss
-- **Optimizers:** Adam, SGD, **LION**  
-- **Schedulers:** StepLR, LambdaLR, **ReduceLROnPlateau**
+- **Optimizers:** Adam, SGD, LION, **AdamW**  
+- **Schedulers:** StepLR, LambdaLR, ReduceLROnPlateau
 - **Regularization:** Dropout, BatchNorm2D  
 - **Utilities:** zeros, randomMatrix, softmax, crossEntropy, dot, addMatrices, reshape, stack, flatten, eye, concat  
 - **Model Container:** Sequential (for stacking layers with forward/backward passes)  
 
+# Others
+
+- **Tokenizer**
+- **Softmax Layer**
+
 ---
 
 ## Installation
@@ -105,7 +109,7 @@ loadModel(model2, json);
 Check the `demo/` directory for ready-to-run demos:
 - **demo/MakeModel.js:** Build and run a simple neural network.
 - **demo/scheduler.js:** Experiment with learning rate schedulers.
-- **demo/fu_fun.js:** Tests All fu_functions (for users). 
+- **demo/fu_fun.js:** Test all user-friendly (fu or For Users/Friendly Users) functions
 - Add your own scripts for quick prototyping!
 
 ```bash
@@ -129,4 +133,4 @@ node demo/fu_fun.js
 
 `MIT License`
 
-**Copyright (c) 2025 rizal-editors**
+**Copyright (c) 2024 rizal-editors**

package/demo/scheduler.js

@@ -1,4 +1,4 @@
-// Example: Test ALL learning rate schedulers with mini-jstorch optimizers
+// Example: Test ALL learning rate schedulers in mini-jstorch with mini-jstorch optimizers
 
 import { SGD, StepLR, LambdaLR, ReduceLROnPlateau, Tensor } from "../src/jstorch.js";
 

package/index.js

@@ -1,2 +1,3 @@
 // package root
+// * = all exports from src/jstorch.js
 export * from "./src/jstorch.js";

package/package.json

@@ -1,22 +1,24 @@
 {
   "name": "mini-jstorch",
-  "version": "1.6.0",
+  "version": "1.7.0",
   "type": "module", 
-  "description": "A lightweight JavaScript neural network library for rapid frontend AI experimentation on low-resource devices Inspired by PyTorch.",
+  "description": "A lightweight JavaScript neural network library for learning AI concepts and rapid Frontend experimentation. PyTorch-inspired, zero dependencies, perfect for educational use.",
   "main": "index.js",
   "keywords": [
     "neural-network",
     "javascript",
     "lightweight-torch",
     "lightweight",
-    "small",
+    "small-torch",
     "javascript-torch",
-    "ai",
+    "ai-model",
     "jstorch",
     "pytorch",
     "front-end",
     "machine-learning",
-    "mini"
+    "tiny-ml",
+    "frontend-ai",
+    "mini-neural-network"
   ],
   "author": "Rizal",
   "license": "MIT",

package/src/jstorch.js

@@ -1,6 +1,6 @@
 /*!
  * Project: mini-jstorch
- * File: MainEngine.js
+ * File: jstorch.js
  * Author: Rizal-editors
  * License: MIT
  * Copyright (C) 2025 Rizal-editors
@@ -24,8 +24,7 @@
  * SOFTWARE.
  */
 
-
-// ---------------------- Utilities ----------------------
+// ---------------------- DONOT USE THESE (ENGINE INTERNALS) ----------------------
 export function zeros(rows, cols) { 
     return Array.from({length:rows},()=>Array(cols).fill(0)); 
 }
@@ -75,7 +74,7 @@ export function crossEntropy(pred,target){
     return -target.reduce((sum,t,i)=>sum+t*Math.log(pred[i]+eps),0); 
 }
 
-// ---------------------- USERS FRIENDLY UTILS ----------------
+// ---------------------- USERS FRIENDLY UTILS (USE THIS!) ----------------
 export function fu_tensor(data, requiresGrad = false) {
     if (!Array.isArray(data) || !Array.isArray(data[0])) {
         throw new Error("fu_tensor: Data must be 2D array");
@@ -592,6 +591,148 @@ export class ReLU{
     }
 }
 
+// ---------------------- Softmax ----------------------
+export class Softmax {
+    constructor(dim = -1) {
+        this.dim = dim;
+        this.output = null;
+        this.input = null;
+    }
+
+    forward(x) {
+        this.input = x;
+        
+        // x: [batch_size, num_classes]
+        this.output = x.map(row => {
+            const maxVal = Math.max(...row);
+            const exps = row.map(v => Math.exp(v - maxVal));
+            const sumExps = exps.reduce((a, b) => a + b, 0);
+            return exps.map(v => v / sumExps);
+        });
+        return this.output;
+    }
+
+    backward(grad) {
+        // grad: [batch_size, num_classes] - gradient from next layer
+        const batchSize = grad.length;
+        const numClasses = grad[0].length;
+        
+        const gradInput = zeros(batchSize, numClasses);
+        
+        for (let i = 0; i < batchSize; i++) {
+            const s = this.output[i]; // Softmax output for this sample
+            const gradOut = grad[i];  // Gradient from loss
+            
+            // Compute Jacobian matrix: J_ij = s_i * (δ_ij - s_j)
+            for (let j = 0; j < numClasses; j++) {
+                let sum = 0;
+                for (let k = 0; k < numClasses; k++) {
+                    // J[j][k] = s[j] * ((j === k ? 1 : 0) - s[k])
+                    const jacobian = s[j] * ((j === k ? 1 : 0) - s[k]);
+                    sum += jacobian * gradOut[k];
+                }
+                gradInput[i][j] = sum;
+            }
+        }
+        
+        return gradInput;
+    }
+
+    parameters() {
+        return []; // Softmax has no trainable parameters
+    }
+}
+
+// ---------------------- Tokenizer ----------------------
+export class Tokenizer {
+    constructor(vocabSize = 2000){
+        this.vocabSize = vocabSize;
+        this.wordToIndex = new Map();
+        this.indexToWord = new Map();
+        this.fitted = false;
+    }
+
+    fit(texts){
+        const wordCounts = new Map();
+
+        // Count word frequencies from all texts
+        texts.forEach(text => {
+            const words = this._preprocess(text);
+            words.forEach(word => {
+                wordCounts.set(word, (wordCounts.get(word) || 0) + 1);
+            });
+        });
+
+        // Sort by frequency and take top words
+        const sortedWords = [...wordCounts.entries()]
+            .sort((a, b) => a[1] - a[1])
+            .slice(0, this.vocabSize - 1); // Reverse 1 for unknown
+
+        // Build vocabulary
+        this.wordToIndex.clear();
+        this.indexToWord.clear();
+
+        // Add unk token
+        this.wordToIndex.set('<UNK>', 0);
+        this.indexToWord.set(0, '<UNK>');
+
+        // Add most frequent words
+        sortedWords.forEach(([word], index) =>{
+            this.wordToIndex.set(word, index + 1);
+            this.indexToWord.set(index + 1, word);
+        })
+
+        this.fitted = true;
+        return this;
+    }
+
+    tokenize(text){
+        if (!this.fitted) throw new Error("Tokenizer not fitted. Call fit() first.");
+
+        const words = this._preprocess(text);
+        return words.map(word => this.wordToIndex.get(word) || 0);
+    }
+
+    tokenizeBatch(texts, maxLength=null){
+        if (!this.fitted) throw new Error("Tokenizer not fitted. Call fit() first.");
+
+        return texts.map(text => {
+            const tokens = this.tokenize(text);
+
+            if (maxLength !== null){
+                // Pad or truncate to maxLength
+                if (tokens.length > maxLength){
+                    return tokens.slice(0, maxLength);
+                } else {
+                    return [...tokens, ...Array(maxLength - tokens.length).fill(0)];
+                }
+            }
+
+            return tokens;
+        })
+    }
+
+    detokenize(tokens){
+        return tokens.map(token => this.indexToWord.get(token) || '<UNK>').join(' ');
+    }
+
+    detokenizeBatch(tokenBatches){
+        return tokenBatches.map(tokens => this.detokenize(tokens));
+    }
+
+    getVocabSize(){
+        return this.wordToIndex.size;
+    }
+
+    _preprocess(text) {
+        return text.toLowerCase()
+                  .replace(/[^\w\s]/g, ' ')  // Remove punctuation
+                  .split(/\s+/)             // Split by whitespace
+                  .filter(word => word.length > 0); // Remove empty strings
+    }
+}
+
+// I'm too lazy to break lines here, so everything stays in one line
 export class Sigmoid{ constructor(){ this.out=null; } forward(x){ const fn=v=>1/(1+Math.exp(-v)); this.out=x.map(r=>r.map(fn)); return this.out; } backward(grad){ return grad.map((r,i)=>r.map((v,j)=>v*this.out[i][j]*(1-this.out[i][j]))); } }
 export class Tanh{ constructor(){ this.out=null; } forward(x){ this.out=x.map(r=>r.map(v=>Math.tanh(v))); return this.out; } backward(grad){ return grad.map((r,i)=>r.map((v,j)=>v*(1-this.out[i][j]**2))); } }
 export class LeakyReLU{ constructor(alpha=0.01){ this.alpha=alpha; this.out=null; } forward(x){ this.out=x.map(r=>r.map(v=>v>0?v:v*this.alpha)); return this.out; } backward(grad){ return grad.map((r,i)=>r.map((v,j)=>v*(this.out[i][j]>0?1:this.alpha))); } }
@@ -664,6 +805,70 @@ export class Adam{
     }
 }
 
+// ---------------------- AdamW Optimizer ----------------------
+export class AdamW {
+    constructor(params, options = {}) {
+        const {
+            lr = 0.001,
+            beta1 = 0.9,
+            beta2 = 0.999,
+            eps = 1e-8,
+            weight_decay = 0.01,  
+            max_grad_norm = 1.0
+        } = options;
+        
+        this.params = params;
+        this.lr = lr;
+        this.beta1 = beta1;
+        this.beta2 = beta2;
+        this.eps = eps;
+        this.weight_decay = weight_decay;
+        this.max_grad_norm = max_grad_norm;
+        
+        this.m = params.map(p => zeros(p.param.length, p.param[0].length || 1));
+        this.v = params.map(p => zeros(p.param.length, p.param[0].length || 1));
+        this.t = 0;
+    }
+
+    step() {
+        this.t++;
+        this.params.forEach((p, idx) => {
+            // Gradient clipping (same as Adam)
+            let grad_norm_sq = 0;
+            for (let i = 0; i < p.param.length; i++) {
+                for (let j = 0; j < (p.param[0].length || 1); j++) {
+                    const grad_val = p.grad[i] && p.grad[i][j] !== undefined ? p.grad[i][j] : 0;
+                    grad_norm_sq += grad_val * grad_val;
+                }
+            }
+            const grad_norm = Math.sqrt(grad_norm_sq);
+            const clip_scale = grad_norm > this.max_grad_norm ? this.max_grad_norm / grad_norm : 1.0;
+
+            // AdamW update: weight decay applied separately
+            for (let i = 0; i < p.param.length; i++) {
+                for (let j = 0; j < (p.param[0].length || 1); j++) {
+                    if (p.grad[i] && p.grad[i][j] !== undefined) {
+                        const g = p.grad[i][j] * clip_scale;
+                        
+                        // Adam moments
+                        this.m[idx][i][j] = this.beta1 * this.m[idx][i][j] + (1 - this.beta1) * g;
+                        this.v[idx][i][j] = this.beta2 * this.v[idx][i][j] + (1 - this.beta2) * g * g;
+                        
+                        const mHat = this.m[idx][i][j] / (1 - Math.pow(this.beta1, this.t));
+                        const vHat = this.v[idx][i][j] / (1 - Math.pow(this.beta2, this.t));
+                        
+                        // AdamW key difference: weight decay applied to weights, not gradients
+                        p.param[i][j] -= this.lr * (
+                            mHat / (Math.sqrt(vHat) + this.eps) + 
+                            this.weight_decay * p.param[i][j]  // Decoupled weight decay
+                        );
+                    }
+                }
+            }
+        });
+    }
+}
+
 export class SGD{
     constructor(params, lr = 0.01, max_grad_norm = 1.0) {
         this.params = params;