mini-jstorch 1.8.1 → 2.0.0

package/Docs/About.md

@@ -1,83 +1,84 @@
-# Mini-JSTorch — Technical Information
-
----
-
-## General Information
-
-- **Project Name:** mini-jstorch  
-- **Internal Name:** JST (JST-orch)
-
-> Note:  
-> Early versions of JST do not strictly follow semantic versioning conventions  
-> (e.g. `0.0.1` for patches, `0.1.0` for minor releases, `1.0.0` for major releases).  
-> This inconsistency reflects the early learning and experimental phase of the project.
-
----
-
-## 1. Engine Architecture Limitations (JST Core)
-
-This section outlines the known structural weaknesses of the JST engine.  
-Although the architecture may appear complex, it is currently sensitive and tightly coupled.
-
-### Identified Limitations
-
-- **High dependency on Utilities**  
-  Every core class depends directly on the Utilities module, which is defined at the top of the `jstorch.js` file. This creates strong coupling across the engine.
-
-- **Limited Tensor dimensionality**  
-  Tensor implementations currently support only two dimensions.  
-  Extending support to higher-dimensional tensors would require significant architectural changes due to the existing complexity.
-
-- **Uneven class complexity**  
-  New or recently modified classes often become significantly more complex than others, leading to inconsistency in maintainability and internal design balance.
-
----
-
-## 2. Rationale Behind the `fu_` Utilities
-
-This section explains why the `fu_` utilities were introduced despite the existence of internal Utilities.
-
-### Issues with Internal Utilities
-
-- The Utilities defined at the beginning of `jstorch.js` are **internal engine helpers**, not intended for direct user interaction.
-
-- These Utilities are heavily reused across multiple core classes.  
-  Any modification to a utility function may trigger **cascading (domino) errors** throughout the engine due to tight dependencies.
-
-- Some utility functions intentionally diverge from standard or expected formulas.  
-  For example:
-  - Expected formula:  
-    `Param1 - Param4 * Param3`
-  - Internal Utilities implementation:  
-    `Param1 - Param2 * Param3 + Param4`
-
-  This behavior exists because internal Utilities are optimized for class-level computations, not for user-facing correctness or predictability.
-
-### Purpose of `fu_` Utilities
-
-The `fu_` utilities were designed to improve the **user experience** by providing:
-
-- Predictable and correct computational behavior
-- User-friendly and stable helper functions
-- Isolation from internal engine changes
-- Reduced risk of incorrect outputs and dependency-based cascading errors
-
-In short, `fu_` exists to ensure safety, clarity, and consistency for end users of Mini-JSTorch.
-
----
-
-## 3. SJK (Shortcut JST Keywords) Reference
-
-This section lists commonly used abbreviations and keywords within the mini-jstorch ecosystem.
-
-**Format:** `"KEYWORD" : "Full Name / Meaning"`
-
-- `"JST"` : JSTorch  
-- `"fu"` : For User / User-Friendly  
-- `"fun"` : Function  
-- `"Dummy"` : Experimental  
-- `"Exp"` : Restricted experimental entity  
-- `"msg"` : Message, comment, warning, announcement  
-- `"donot"` : Do not / Don't  
-
----
+# Mini-JSTorch — Technical Information
+
+---
+
+## General Information
+
+- **Project Name:** mini-jstorch  
+- **Internal Name:** JST (JST-orch)
+
+> Note:  
+> Early versions of JST do not strictly follow semantic versioning conventions  
+> (e.g. `0.0.1` for patches, `0.1.0` for minor releases, `1.0.0` for major releases).  
+> This inconsistency reflects the early learning and experimental phase of the project.
+
+---
+
+## 1. Engine Architecture Limitations (JST Core)
+
+This section outlines the known structural weaknesses of the JST engine.  
+Although the architecture may appear complex, it is currently sensitive and tightly coupled.
+
+### Identified Limitations
+
+- **High dependency on Utilities**  
+  Every core class depends directly on the Utilities module, which is defined at the top of the `jstorch.js` file. This creates strong coupling across the engine.
+
+- **Limited Tensor dimensionality**  
+  Tensor implementations currently support only two dimensions.  
+  Extending support to higher-dimensional tensors would require significant architectural changes due to the existing complexity.
+
+- **Uneven class complexity**  
+  New or recently modified classes often become significantly more complex than others, leading to inconsistency in maintainability and internal design balance.
+
+---
+
+## 2. Rationale Behind the `fu_` Utilities
+
+This section explains why the `fu_` utilities were introduced despite the existence of internal Utilities.
+
+### Issues with Internal Utilities
+
+- The Utilities defined at the beginning of `jstorch.js` are **internal engine helpers**, not intended for direct user interaction.
+
+- These Utilities are heavily reused across multiple core classes.  
+  Any modification to a utility function may trigger **cascading (domino) errors** throughout the engine due to tight dependencies.
+
+- Some utility functions intentionally diverge from standard or expected formulas.  
+  For example:
+  - Expected formula:  
+    `Param1 - Param4 * Param3`
+  - Internal Utilities implementation:  
+    `Param1 - Param2 * Param3 + Param4`
+
+  This behavior exists because internal Utilities are optimized for class-level computations, not for user-facing correctness or predictability.
+
+### Purpose of `fu_` Utilities
+
+The `fu_` utilities were designed to improve the **user experience** by providing:
+
+- Predictable and correct computational behavior
+- User-friendly and stable helper functions
+- Isolation from internal engine changes
+- Reduced risk of incorrect outputs and dependency-based cascading errors
+
+In short, `fu_` exists to ensure safety, clarity, and consistency for end users of Mini-JSTorch.
+
+---
+
+## 3. SJK (Shortcut JST Keywords) Reference
+
+This section lists commonly used abbreviations and keywords within the mini-jstorch ecosystem.
+
+**Format:** `"KEYWORD" : "Full Name / Meaning"`
+
+- `"JST"` : JSTorch  
+- `"fu"` : For User / User-Friendly  
+- `"fun"` : Function  
+- `"Dummy"` : Experimental  
+- `"Exp"` : Restricted experimental entity  
+- `"msg"` : Message, comment, warning, announcement  
+- `"donot"` : Do not / Don't  
+// add more.
+
+---

package/Docs/Structure.md

@@ -1,129 +1,116 @@
-# Project File Structure # 
-
-This document describes the directory and file structure of the **mini-JSTorch** package.
-It provides an overview of how the project is organized and the purpose of each major component.
-
----
-
-## Repository Overview
-
-```text
-mini-jstorch/
-├── demo/
-│   ├── fu_fun.js
-│   ├── MakeModel.js
-│   └── scheduler.js
-├── Docs/
-│   ├── About.md
-│   └── Structure.md
-├── src/
-│   ├── jstorch.js
-│   └── Dummy/
-│       └── msg/
-├── index.js
-├── package.json
-└── README.md
-```
-
----
-
-## Directory Descriptions
-
-`/demo`
-
-- Contains demonstration and testing files.
-
-  - Used for unit testing, quick system checks, and example usage
-  - Intended for users who prefer practical examples over reading full API documentation
-  - Allows testing features without writing extensive manual code
-
-`/Docs`
-
-- Contains detailed documentation related to the mini-JSTorch package.
-
-  - Provides deeper explanations of internal design and usage
-  - Intended for contributors and advanced users
-
-`/src`
-
-- Contains the source code of the JSTorch engine.
-
-  - Houses all core logic and internal implementations
-  - Modifications in this directory directly affect engine behavior
-
-`/src/Dummy`
-
-- Experimental and restricted directory.
-
-  - Used for experimental purposes and future development
-  - Files inside this directory may be unstable or incomplete
-  - Not intended for public or production use
-
-`/src/Dummy/msg`
-
-- Contains warning or message files.
-
-  - Indicates that files within the `Dummy` directory are restricted
-  - Serves as a notification mechanism for experimental or future-update-related content
-
----
-
-## File Descriptions
-
-`/demo/fu_fun.js`
-
-- Purpose: Tests all user-facing (`fu_`) functions
-- Notes: Focuses on friendly and predictable helper utilities
-
-`/demo/MakeModel.js`
-
-- Purpose: Demonstrates creation of a simple model
-- Notes: Uses the `StepLR` scheduler as part of the example workflow
-
-`/demo/scheduler.js`
-
-- Purpose: Tests scheduler-related functionality
-- Notes: Intended to validate learning rate scheduling behavior
-
-`/Docs/About.md`
-
-- Purpose: Contains additional information about the mini-JSTorch package
-- Notes: May include background, design decisions, or non-API-related explanations
-
-`/Docs/Structure.md`
-
-- Purpose: Documents the repository file and folder structure
-- Notes: This file
-
-`/src/jstorch.js`
-
-- Purpose: Core engine implementation
-
-- Notes:
-
-  - Contains all JSTorch engine logic and functions
-  - Central file of the entire package
-  - Changes here have wide-ranging effects
-
-`index.js`
-
-- Purpose: Package entry point
-- Notes: Exposes public APIs and connects internal modules
-
-`package.json`
-
-- Purpose: Project configuration and metadata
-- Notes: Defines dependencies, scripts, and package information
-
-`README.md`
-
-- Purpose: Main documentation entry
-- Notes: Provides overview, installation instructions, and basic usage
-
-**Notes**
-
-- Experimental files may change or be restricted without notice
-- Users are encouraged to rely on public APIs and documented utilities
-- Internal structures are subject to refactoring as the project evolves
-
+# Project File Structure # 
+
+This document describes the directory and file structure of the **mini-JSTorch** package.
+It provides an overview of how the project is organized and the purpose of each major component.
+
+---
+
+## Repository Overview
+
+```text
+mini-jstorch/
+├── demo/
+│   ├── fu_fun.js
+│   ├── MakeModel.js
+│   └── scheduler.js
+        xor_classification.js
+        linear_regression.js
+├── Docs/
+│   ├── About.md
+│   └── Structure.md
+├── src/
+│   ├── jstorch.js
+│   └── Dummy/
+│       └── msg/
+├── index.js
+├── package.json
+└── README.md
+```
+
+---
+
+## Directory Descriptions
+
+`/demo`
+
+- Contains demonstration and testing files.
+
+  - Used for unit testing, quick system checks, and example usage
+  - Intended for users who prefer practical examples over reading full API documentation
+  - Allows testing features without writing extensive manual code
+
+`/Docs`
+
+- Contains detailed documentation related to the mini-JSTorch package.
+
+  - Provides deeper explanations of internal design and usage
+  - Intended for contributors and advanced users
+
+`/src`
+
+- Contains the source code of the JSTorch engine.
+
+  - Houses all core logic and internal implementations
+  - Modifications in this directory directly affect engine behavior
+
+`/src/Dummy`
+
+- Experimental and restricted directory.
+
+  - Used for experimental purposes and future development
+  - Files inside this directory may be unstable or incomplete
+  - Not intended for public or production use
+
+`/src/Dummy/msg`
+
+- Contains warning or message files.
+
+  - Indicates that files within the `Dummy` directory are restricted
+  - Serves as a notification mechanism for experimental or future-update-related content
+
+---
+
+## File Descriptions
+
+`/Docs/About.md`
+
+- Purpose: Contains additional information about the mini-JSTorch package
+- Notes: May include background, design decisions, or non-API-related explanations
+
+`/Docs/Structure.md`
+
+- Purpose: Documents the repository file and folder structure
+- Notes: This file
+
+`/src/jstorch.js`
+
+- Purpose: Core engine implementation
+
+- Notes:
+
+  - Contains all JSTorch engine logic and functions
+  - Central file of the entire package
+  - Changes here have wide-ranging effects
+
+`index.js`
+
+- Purpose: Package entry point
+- Notes: Exposes public APIs and connects internal modules
+
+`package.json`
+
+- Purpose: Project configuration and metadata
+- Notes: Defines dependencies, scripts, and package information
+
+`README.md`
+
+- Purpose: Main documentation entry
+- Notes: Provides overview, installation instructions, and basic usage
+
+**Notes**
+
+- Experimental files may change or be restricted without notice
+- Users are encouraged to rely on public APIs and documented utilities
+- Internal structures are subject to refactoring as the project evolves
+
 ---

package/README.md

@@ -1,12 +1,26 @@
-## Mini-JSTorch
+## Mini-JSTorch (MAJOR UPDATE)
 
+---
 
 Mini-JSTorch is a lightweight, `dependency-free` JavaScript neural network library designed for `education`, `experimentation`, and `small-scale models`.
 It runs in Node.js and modern browsers, with a simple API inspired by PyTorch-style workflows.
 
 This project prioritizes `clarity`, `numerical correctness`, and `accessibility` over performance or large-scale production use.
 
-In this version `1.8.0`, we Introduce the **SoftmaxCrossEntropyLoss**, and **BCEWithLogitsLoss**
+In this version `2.0.0`, we introduce:
+- **Fixed Linear layer cache** (critical bug fix for training)
+- **Fixed GELU gradient calculation**
+- **Fixed MSELoss gradient scaling**
+- **Optimized Softmax gradient** (O(n²) → O(n))
+- **Improved Tokenizer** with proper PAD/UNK separation
+- **Added Sequential.zeroGrad(), train(), eval(), stateDict() methods**
+
+---
+
+**⚠️ BREAKING CHANGES in v2.0.0:**
+- Tokenizer API: `tokenizeBatch()` → `transform()`, `detokenizeBatch()` → `inverseTransform()`
+- Tokenizer now uses `<PAD>` at index 0 and `<UNK>` at index 1
+- MSELoss gradient scale now matches PyTorch behavior
 
 ---
 
@@ -21,16 +35,13 @@ In this version `1.8.0`, we Introduce the **SoftmaxCrossEntropyLoss**, and **BCE
 - running simple training loops in the browser
 - environments where large frameworks are unnecessary or unavailable
 
-`Mini-JSTorch is NOT a replacement for PyTorch, TensorFlow, or TensorFlow.js.`
-
-`It is intentionally scoped to remain small, readable, and easy to debug.`
+`mini-jstorch is intentionally designed to be small, readable, and easy to debug.`
 
 ---
 
 # Key Characteristics
 
 - Zero dependencies
-- ESM-first (`type: module`)
 - Works in Node.js or others enviornments and browser environments
 - Explicit, manual forward and backward passes
 - Focused on 2D training logic (`[batch][features]`)
@@ -135,7 +146,7 @@ In Browser/Website:
 # Loss Functions
 
 - MSELoss
-- CrossEntropyLoss (*legacy*)
+- CrossEntropyLoss (*legacy*, use **SoftmaxCrossEntropy** instead)
 - SoftmaxCrossEntropyLoss (**recommended**)
 - BCEWithLogitsLoss (**recommended**)
 
@@ -152,7 +163,7 @@ In Browser/Website:
 - LambdaLR
 - ReduceLROnPlateau
 - Regularization
-- Dropout (*basic*, *educational*)
+- Dropout
 - BatchNorm2D (*experimental*)
 
 # Utilities
@@ -203,9 +214,9 @@ import {
 } from "./src/jstorch.js";
 
 const model = new Sequential([
-  new Linear(2, 4),
+  new Linear(2, 8),
   new ReLU(),
-  new Linear(4, 2) // logits output
+  new Linear(8, 2) // logits output
 ]);
 
 const X = [
@@ -217,7 +228,7 @@ const Y = [
 ];
 
 const lossFn = new SoftmaxCrossEntropyLoss();
-const optimizer = new Adam(model.parameters(), 0.1);
+const optimizer = new Adam(model.parameters(), {lr: 0.1});
 
 for (let epoch = 1; epoch <= 300; epoch++) {
   const logits = model.forward(X);
@@ -225,13 +236,16 @@ for (let epoch = 1; epoch <= 300; epoch++) {
   const grad = lossFn.backward();
   model.backward(grad);
   optimizer.step();
+  
+  // Zero gradients for next iteration
+  model.zeroGrad();
 
   if (epoch % 50 === 0) {
-    console.log(`Epoch ${epoch}, Loss: ${loss.toFixed(4)}`);
+    console.log(`Epoch ${epoch}, Loss: ${loss.toFixed(6)}`);
   }
 }
 ```
-Do not combine `SoftmaxCrossEntropyLoss` with a `Softmax` layer.
+`Important:` Do not combine `SoftmaxCrossEntropyLoss` with a `Softmax` layer.
 
 # Binary Classifiaction (BCEWithLogitsLoss)
 
@@ -245,21 +259,20 @@ import {
 } from "./src/jstorch.js";
 
 const model = new Sequential([
-  new Linear(2, 4),
+  new Linear(2, 8),
   new ReLU(),
-  new Linear(4, 1) // logit
+  new Linear(8, 1) // logit output
 ]);
 
 const X = [
   [0,0], [0,1], [1,0], [1,1]
 ];
-
 const Y = [
   [0], [1], [1], [0]
 ];
 
 const lossFn = new BCEWithLogitsLoss();
-const optimizer = new Adam(model.parameters(), 0.1);
+const optimizer = new Adam(model.parameters(), {lr: 0.1});
 
 for (let epoch = 1; epoch <= 300; epoch++) {
   const logits = model.forward(X);
@@ -267,7 +280,37 @@ for (let epoch = 1; epoch <= 300; epoch++) {
   const grad = lossFn.backward();
   model.backward(grad);
   optimizer.step();
+  model.zeroGrad();
+  
+  // Print progress every 50 epochs
+  if (epoch % 50 === 0) {
+    const probs = logits.map(p => 1 / (1 + Math.exp(-p[0])));
+    console.log(`Epoch ${epoch} | Loss: ${loss.toFixed(6)}`);
+    probs.forEach((prob, i) => {
+      const pred = prob > 0.5 ? 1 : 0;
+      console.log(`  [${X[i]}] → prob: ${prob.toFixed(4)} (${pred}) | target: ${Y[i][0]}`);
+    });
+    console.log('');
+  }
 }
+
+// Final evaluation
+console.log("\nTraining Complete\n");
+model.eval(); 
+
+const finalLogits = model.forward(X);
+const finalProbs = finalLogits.map(p => 1 / (1 + Math.exp(-p[0])));
+
+console.log("Final Results:");
+let correct = 0;
+finalProbs.forEach((prob, i) => {
+  const pred = prob > 0.5 ? 1 : 0;
+  const target = Y[i][0];
+  const isCorrect = pred === target;
+  if (isCorrect) correct++;
+  console.log(`  [${X[i]}] → ${prob.toFixed(4)} (${pred}) | target: ${target} ${isCorrect ? '✓' : '✗'}`);
+});
+console.log(`\nAccuracy: ${(correct / X.length * 100).toFixed(2)}%`);
 ```
 Do not combine `BCEWithLogitsLoss` with a `Sigmoid` layer.
 
@@ -276,7 +319,8 @@ Do not combine `BCEWithLogitsLoss` with a `Sigmoid` layer.
 # Save & Load Models 
 
 ```javascript
-import { saveModel, loadModel, Sequential } from "mini-jstorch";
+// WARN: Error/Bug may be expected for this time!
+import { saveModel, loadModel, Sequential } from "./src/jstorch.js";
 
 const json = saveModel(model);
 const model2 = new Sequential([...]); // same architecture
@@ -287,16 +331,18 @@ loadModel(model2, json);
 
 # Demos
 
-See the `demo/` directory for runnable examples:
-- `demo/MakeModel.js` – simple training loop
-- `demo/scheduler.js` – learning rate schedulers
-- `demo/fu_fun.js` – utility functions
+See the `demo/` directory for runnable examples!
+- `demo/fu_fun.js`
+- `demo/MakeModel.js`
+- `demo/scheduler.js`
+- `demo/xor_classification.js`
+- `demo/linear_regression.js`
+
 
 ```bash
-node demo/MakeModel.js
-node demo/scheduler.js
-node demo/fu_fun.js
+node demo/<fileNameInDemo>.js
 ```
+**Make sure your directory while run this at root folder!**
 
 ---
 
@@ -310,15 +356,6 @@ node demo/fu_fun.js
 
 ---
 
-# Intended Use Cases
-
-- Learning how neural networks work internally
-- Teaching ML fundamentals
-- Small experiments in Node.js or the browser
-- Lightweight AI demos without GPU or large frameworks
-
---- 
-
 # License
 
 MIT License