adv-optm 1.2.dev9__tar.gz → 2.1.0__tar.gz

{adv_optm-1.2.dev9 → adv_optm-2.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: adv_optm
-Version: 1.2.dev9
+Version: 2.1.0
 Summary: A family of highly efficient, lightweight yet powerful optimizers.
 Home-page: https://github.com/Koratahiu/Advanced_Optimizers
 Author: Koratahiu
@@ -35,6 +35,32 @@ A comprehensive, all-in-one collection of optimization algorithms for deep learn
 
 [![PyPI](https://img.shields.io/pypi/v/adv_optm)](https://pypi.org/project/adv_optm/)
 
+## 🔥 What's New
+
+### in 2.0.x
+
+* Implemented torch.compile for all advanced optimizers. Enabled via (compiled_optimizer=True) to fuse and optimize the optimizer step path.
+* Better and improved 1-bit factored mode via (nnmf_factor=True).
+* Various improvements across the optimizers.
+
+### in 1.2.x
+* Added **advanced variants** of [Muon optimizer](https://kellerjordan.github.io/posts/muon/) with **features** and **settings** from recent papers.
+
+| Optimizer | Description |
+|---|---|
+| `Muon_adv` | Advanced Muon implementation with CANS, NorMuon, Low-Rank ortho, etc. features. |
+| `AdaMuon_adv` | Advanced AdaMuon implementation, which combines Muon's geometry with Adam-like adaptive scaling and sign-based orthogonalization. |
+
+> *Documentation coming soon.*
+
+* Implemented [Cautious Weight Decay](https://arxiv.org/abs/2510.12402) for all advanced optimizers.
+
+* Improved parameter update and weight decay for **BF16** with **stochastic rounding**. The updates are now accumulated in **float32** and rounded once at the end.
+
+* Use fused and in-place operations whenever possible for all advanced optimizers.
+
+* **Prodigy variants** are now **50% faster** by [avoiding CUDA syncs](https://github.com/Koratahiu/Advanced_Optimizers/pull/5). Thanks to **@dxqb**!
+
 ---
 
 ## 📦 Installation
@@ -52,7 +78,7 @@ This library integrates multiple state-of-the-art optimization techniques valida
 ### **Memory-Efficient Optimization (SMMF-inspired)**
 - **Paper**: [SMMF: Square-Matricized Momentum Factorization](https://arxiv.org/abs/2412.08894)
 - **Approach**: Uses rank-1 non-negative matrix factorization with reconstruction cycle (factor → reconstruct → update → factor)
-- **Innovation**: 
+- **Innovation**:
   - First moment split into **1-bit sign + absolute value**
   - Final storage: **four factored vectors + one 1-bit sign state**
   - Preserves Adam-like update quality with drastically reduced memory
@@ -110,7 +136,7 @@ This library integrates multiple state-of-the-art optimization techniques valida
 
 ## 🛠️ Comprehensive Feature Guide
 
-### A. Universal Safe Features  
+### A. Universal Safe Features
 *These features work with all optimizers and are generally safe to enable.*
 
 | Feature | Description | Recommended Usage | Performance Impact | Theoretical Basis | Compatibility |
@@ -141,7 +167,6 @@ This library integrates multiple state-of-the-art optimization techniques valida
 
 - Adds a **slow-decaying second EMA** (`beta3`) that retains gradient memory over tens of thousands of steps.
 - Particularly effective for **small batch sizes**, where Adam’s standard first moment is nearly useless.
-- **Reference**: [AdaMeM: Memory Efficient Momentum for Adafactor](https://openreview.net/forum?id=fZqMVTz7K5)
 
 #### Tunable Hyperparameters
 | Parameter | Default | Tuning Guide |
@@ -156,7 +181,7 @@ This library integrates multiple state-of-the-art optimization techniques valida
 ### Simplified_AdEMAMix
 
 - Introduced in [Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants (arXiv:2502.02431)](https://arxiv.org/abs/2502.02431).
-- Replaces Adam’s first moment with a **gradient accumulator**, combining the stability of long memory with responsiveness to recent gradients.
+- Replaces Adam’s first moment with a **theory-based momentum** with emphasize on raw gradient, combining the stability of long memory with responsiveness to recent gradients.
 - **Key insight**: Classical momentum **does not accelerate** in noisy (small-batch) regimes; this accumulator do.
 
 #### Tunable Hyperparameters
@@ -165,26 +190,13 @@ This library integrates multiple state-of-the-art optimization techniques valida
 | `beta1` | 0.99 | Controls accumulator memory length:<br>• Small BS: **0.99–0.9999**<br>• Large BS: **0.9** |
 | `Grad α` | 100 | Most critical parameter:<br>• Inversely scales with batch size<br>• **100–10** for small BS (≤32)<br>• **1–0.1** for large BS (≥512) |
 
-> ⚠️ **Critical**: Requires **~100x smaller learning rate** than AdamW (e.g., 1e-6 vs 1e-4).  
+> ⚠️ **Critical**: Requires **~100x smaller learning rate** than AdamW (e.g., 1e-6 vs 1e-4).
 > For `Prodigy_Adv`, set `initial_d` to:
 > - **LoRA**: `1e-8`
 > - **Full FT**: `1e-10`
 > - **Embedding**: `1e-7`
 
-> ⚠️ **Incompatible** with: **Cautious**, **Grams**, **atan2**, and standard gradient clipping.
-
-#### Performance Validation
-
-**Small Batch Training (SDXL, BS=2, 1.8K steps)**  
-![Training Comparison](https://github.com/user-attachments/assets/7eff0671-cc59-47fc-8b63-d5205456d649)
-
-- **🟢 Prodigy_Adv** (beta1=0.9, d0=1e-5): Final LR = 2.9e-4  
-- **🔵 Prodigy_Adv + Simplified_AdEMAMix** (beta1=0.99, α=100, d0=1e-7): Final LR = 5.8e-6
-
-**Results**:
-- Faster convergence and higher final performance with Simplified_AdEMAMix
-- D-Adaptation automatically compensates for aggressive updates
-- Generated samples show **significantly better quality**
+> ⚠️ **Incompatible** with: **Cautious**, **Grams**, **atan2**, and standard update clipping.
 
 ---
 
@@ -194,6 +206,10 @@ This library integrates multiple state-of-the-art optimization techniques valida
 - Automatically clips updates to **[-2, 2]**, preventing destabilizing jumps.
 - **Highly recommended** for `Adopt_Adv`, which is prone to instability without clipping.
 
+> 📚 **Reference**:
+> - Paper: https://arxiv.org/abs/2407.05872
+> - Code: https://github.com/lucidrains/adam-atan2-pytorch
+
 ---
 
 ### **Kourkoutas-β**
@@ -202,8 +218,8 @@ This library integrates multiple state-of-the-art optimization techniques valida
 
 Instead of using a fixed β₂ (e.g., 0.999 or 0.95), it **dynamically modulates β₂ per layer** based on a bounded *sunspike ratio*:
 
-- **During gradient bursts** → β₂ ↓ toward `Lower β₂` → faster reaction  
-- **During calm phases** → β₂ ↑ toward `The Selected β₂` → stronger smoothing  
+- **During gradient bursts** → β₂ ↓ toward `Lower β₂` → faster reaction
+- **During calm phases** → β₂ ↑ toward `The Selected β₂` → stronger smoothing
 
 This is especially effective for **noisy training, small batch sizes, and high learning rates**, where gradient norms shift abruptly due to noise or aggressive LR schedules.
 
@@ -216,60 +232,17 @@ This is especially effective for **noisy training, small batch sizes, and high l
 
 > 💡 **Best Practice**: Set `K_warmup_steps` equal to your standard LR warmup steps. During warmup, the optimizer uses the static `beta2`; adaptation begins only after warmup ends.
 
-> 🔍 **Debugging Aid**: Enable `K_Logging` to monitor (min, max, mean) of dynamic β₂ values across layers every *N* steps.
-
-#### 📊 Performance Validation
-
-**ADAMW_ADV - full SDXL finetuning (aggressive LR: 3e-5) (BS=4, 2.5K steps)**  
-<img width="1460" height="382" alt="image" src="https://github.com/user-attachments/assets/007f278a-fbac-4f3d-9cc7-274c3b959cdd" />
-
-- 🟣 Fixed `beta2=0.999`  
-- 🟠 Auto K-beta  
-
-**Observations:**  
-- K-beta is clearly better and more robust/stable for high LRs.
-
-> 📚 **Reference**:  
-> - Paper: [Kourkoutas-β: A Sunspike-Driven Adam Optimizer with Desert Flair](https://arxiv.org/abs/2508.12996)  
+> 📚 **Reference**:
+> - Paper: [Kourkoutas-β: A Sunspike-Driven Adam Optimizer with Desert Flair](https://arxiv.org/abs/2508.12996)
 > - Code: [kbeta](https://github.com/sck-at-ucy/kbeta)
 
 ---
 
-## Recommended Preset (Tested on LoRA/FT/Embedding)
-
-```yaml
-Learning Rate: 1
-optimizer: PRODIGY_Adv
-settings:
-  - beta1: 0.99 # Controls momentum decay, ~100-step effective memory. Adjust to 0.999 (1000 steps) or 0.9999 (10000 steps) based on training length and stability needs.
-  - beta2: 0.999
-  - kourkoutas_beta: True   # For Kourkoutas-β
-  - K-β Warmup Steps: 50    # Or 100, 200, depending on your run
-  - Simplified_AdEMAMix: True
-  - Grad α: 100
-  - OrthoGrad: True
-  - weight_decay: 0.0
-  - initial_d:
-      • LoRA: 1e-8
-      • Full fine-tune: 1e-10
-      • Embedding: 1e-7
-  - d_coef: 1
-  - d_limiter: True # To stablizie Prodigy with Simplified_AdEMAMix
-  - factored: False  # Can be true or false, quality should not degrade due to Simplified_AdEMAMix’s high tolerance to 1-bit factorization.
-```
-
-> ✅ **Why it works**:  
-> - `Kourkoutas-β` handles beta2 values
-> - `Simplified_AdEMAMix` ensures responsiveness in small-batch noise
-> - `OrthoGrad` prevents overfitting without weight decay
-
----
-
 ## 📚 References
 
-1. [Revisiting BFloat16 Training](https://arxiv.org/abs/2010.06192)  
-2. [SMMF: Square-Matricized Momentum Factorization](https://arxiv.org/abs/2412.08894)  
-3. [The AdEMAMix Optimizer](https://arxiv.org/abs/2409.03137)  
-4. [Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD](https://arxiv.org/abs/2502.02431)  
-5. [AdaMeM: Memory Efficient Momentum for Adafactor](https://openreview.net/forum?id=fZqMVTz7K5)  
+1. [Revisiting BFloat16 Training](https://arxiv.org/abs/2010.06192)
+2. [SMMF: Square-Matricized Momentum Factorization](https://arxiv.org/abs/2412.08894)
+3. [The AdEMAMix Optimizer](https://arxiv.org/abs/2409.03137)
+4. [Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD](https://arxiv.org/abs/2502.02431)
 6. [Kourkoutas-β: A Sunspike-Driven Adam Optimizer with Desert Flair](https://arxiv.org/abs/2508.12996)
+7. [Scaling Exponents Across Parameterizations and Optimizers](https://arxiv.org/abs/2407.05872)

{adv_optm-1.2.dev9 → adv_optm-2.1.0}/README.md

@@ -4,6 +4,32 @@ A comprehensive, all-in-one collection of optimization algorithms for deep learn
 
 [![PyPI](https://img.shields.io/pypi/v/adv_optm)](https://pypi.org/project/adv_optm/)
 
+## 🔥 What's New
+
+### in 2.0.x
+
+* Implemented torch.compile for all advanced optimizers. Enabled via (compiled_optimizer=True) to fuse and optimize the optimizer step path.
+* Better and improved 1-bit factored mode via (nnmf_factor=True).
+* Various improvements across the optimizers.
+
+### in 1.2.x
+* Added **advanced variants** of [Muon optimizer](https://kellerjordan.github.io/posts/muon/) with **features** and **settings** from recent papers.
+
+| Optimizer | Description |
+|---|---|
+| `Muon_adv` | Advanced Muon implementation with CANS, NorMuon, Low-Rank ortho, etc. features. |
+| `AdaMuon_adv` | Advanced AdaMuon implementation, which combines Muon's geometry with Adam-like adaptive scaling and sign-based orthogonalization. |
+
+> *Documentation coming soon.*
+
+* Implemented [Cautious Weight Decay](https://arxiv.org/abs/2510.12402) for all advanced optimizers.
+
+* Improved parameter update and weight decay for **BF16** with **stochastic rounding**. The updates are now accumulated in **float32** and rounded once at the end.
+
+* Use fused and in-place operations whenever possible for all advanced optimizers.
+
+* **Prodigy variants** are now **50% faster** by [avoiding CUDA syncs](https://github.com/Koratahiu/Advanced_Optimizers/pull/5). Thanks to **@dxqb**!
+
 ---
 
 ## 📦 Installation
@@ -21,7 +47,7 @@ This library integrates multiple state-of-the-art optimization techniques valida
 ### **Memory-Efficient Optimization (SMMF-inspired)**
 - **Paper**: [SMMF: Square-Matricized Momentum Factorization](https://arxiv.org/abs/2412.08894)
 - **Approach**: Uses rank-1 non-negative matrix factorization with reconstruction cycle (factor → reconstruct → update → factor)
-- **Innovation**: 
+- **Innovation**:
   - First moment split into **1-bit sign + absolute value**
   - Final storage: **four factored vectors + one 1-bit sign state**
   - Preserves Adam-like update quality with drastically reduced memory
@@ -79,7 +105,7 @@ This library integrates multiple state-of-the-art optimization techniques valida
 
 ## 🛠️ Comprehensive Feature Guide
 
-### A. Universal Safe Features  
+### A. Universal Safe Features
 *These features work with all optimizers and are generally safe to enable.*
 
 | Feature | Description | Recommended Usage | Performance Impact | Theoretical Basis | Compatibility |
@@ -110,7 +136,6 @@ This library integrates multiple state-of-the-art optimization techniques valida
 
 - Adds a **slow-decaying second EMA** (`beta3`) that retains gradient memory over tens of thousands of steps.
 - Particularly effective for **small batch sizes**, where Adam’s standard first moment is nearly useless.
-- **Reference**: [AdaMeM: Memory Efficient Momentum for Adafactor](https://openreview.net/forum?id=fZqMVTz7K5)
 
 #### Tunable Hyperparameters
 | Parameter | Default | Tuning Guide |
@@ -125,7 +150,7 @@ This library integrates multiple state-of-the-art optimization techniques valida
 ### Simplified_AdEMAMix
 
 - Introduced in [Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants (arXiv:2502.02431)](https://arxiv.org/abs/2502.02431).
-- Replaces Adam’s first moment with a **gradient accumulator**, combining the stability of long memory with responsiveness to recent gradients.
+- Replaces Adam’s first moment with a **theory-based momentum** with emphasize on raw gradient, combining the stability of long memory with responsiveness to recent gradients.
 - **Key insight**: Classical momentum **does not accelerate** in noisy (small-batch) regimes; this accumulator do.
 
 #### Tunable Hyperparameters
@@ -134,26 +159,13 @@ This library integrates multiple state-of-the-art optimization techniques valida
 | `beta1` | 0.99 | Controls accumulator memory length:<br>• Small BS: **0.99–0.9999**<br>• Large BS: **0.9** |
 | `Grad α` | 100 | Most critical parameter:<br>• Inversely scales with batch size<br>• **100–10** for small BS (≤32)<br>• **1–0.1** for large BS (≥512) |
 
-> ⚠️ **Critical**: Requires **~100x smaller learning rate** than AdamW (e.g., 1e-6 vs 1e-4).  
+> ⚠️ **Critical**: Requires **~100x smaller learning rate** than AdamW (e.g., 1e-6 vs 1e-4).
 > For `Prodigy_Adv`, set `initial_d` to:
 > - **LoRA**: `1e-8`
 > - **Full FT**: `1e-10`
 > - **Embedding**: `1e-7`
 
-> ⚠️ **Incompatible** with: **Cautious**, **Grams**, **atan2**, and standard gradient clipping.
-
-#### Performance Validation
-
-**Small Batch Training (SDXL, BS=2, 1.8K steps)**  
-![Training Comparison](https://github.com/user-attachments/assets/7eff0671-cc59-47fc-8b63-d5205456d649)
-
-- **🟢 Prodigy_Adv** (beta1=0.9, d0=1e-5): Final LR = 2.9e-4  
-- **🔵 Prodigy_Adv + Simplified_AdEMAMix** (beta1=0.99, α=100, d0=1e-7): Final LR = 5.8e-6
-
-**Results**:
-- Faster convergence and higher final performance with Simplified_AdEMAMix
-- D-Adaptation automatically compensates for aggressive updates
-- Generated samples show **significantly better quality**
+> ⚠️ **Incompatible** with: **Cautious**, **Grams**, **atan2**, and standard update clipping.
 
 ---
 
@@ -163,6 +175,10 @@ This library integrates multiple state-of-the-art optimization techniques valida
 - Automatically clips updates to **[-2, 2]**, preventing destabilizing jumps.
 - **Highly recommended** for `Adopt_Adv`, which is prone to instability without clipping.
 
+> 📚 **Reference**:
+> - Paper: https://arxiv.org/abs/2407.05872
+> - Code: https://github.com/lucidrains/adam-atan2-pytorch
+
 ---
 
 ### **Kourkoutas-β**
@@ -171,8 +187,8 @@ This library integrates multiple state-of-the-art optimization techniques valida
 
 Instead of using a fixed β₂ (e.g., 0.999 or 0.95), it **dynamically modulates β₂ per layer** based on a bounded *sunspike ratio*:
 
-- **During gradient bursts** → β₂ ↓ toward `Lower β₂` → faster reaction  
-- **During calm phases** → β₂ ↑ toward `The Selected β₂` → stronger smoothing  
+- **During gradient bursts** → β₂ ↓ toward `Lower β₂` → faster reaction
+- **During calm phases** → β₂ ↑ toward `The Selected β₂` → stronger smoothing
 
 This is especially effective for **noisy training, small batch sizes, and high learning rates**, where gradient norms shift abruptly due to noise or aggressive LR schedules.
 
@@ -185,60 +201,17 @@ This is especially effective for **noisy training, small batch sizes, and high l
 
 > 💡 **Best Practice**: Set `K_warmup_steps` equal to your standard LR warmup steps. During warmup, the optimizer uses the static `beta2`; adaptation begins only after warmup ends.
 
-> 🔍 **Debugging Aid**: Enable `K_Logging` to monitor (min, max, mean) of dynamic β₂ values across layers every *N* steps.
-
-#### 📊 Performance Validation
-
-**ADAMW_ADV - full SDXL finetuning (aggressive LR: 3e-5) (BS=4, 2.5K steps)**  
-<img width="1460" height="382" alt="image" src="https://github.com/user-attachments/assets/007f278a-fbac-4f3d-9cc7-274c3b959cdd" />
-
-- 🟣 Fixed `beta2=0.999`  
-- 🟠 Auto K-beta  
-
-**Observations:**  
-- K-beta is clearly better and more robust/stable for high LRs.
-
-> 📚 **Reference**:  
-> - Paper: [Kourkoutas-β: A Sunspike-Driven Adam Optimizer with Desert Flair](https://arxiv.org/abs/2508.12996)  
+> 📚 **Reference**:
+> - Paper: [Kourkoutas-β: A Sunspike-Driven Adam Optimizer with Desert Flair](https://arxiv.org/abs/2508.12996)
 > - Code: [kbeta](https://github.com/sck-at-ucy/kbeta)
 
 ---
 
-## Recommended Preset (Tested on LoRA/FT/Embedding)
-
-```yaml
-Learning Rate: 1
-optimizer: PRODIGY_Adv
-settings:
-  - beta1: 0.99 # Controls momentum decay, ~100-step effective memory. Adjust to 0.999 (1000 steps) or 0.9999 (10000 steps) based on training length and stability needs.
-  - beta2: 0.999
-  - kourkoutas_beta: True   # For Kourkoutas-β
-  - K-β Warmup Steps: 50    # Or 100, 200, depending on your run
-  - Simplified_AdEMAMix: True
-  - Grad α: 100
-  - OrthoGrad: True
-  - weight_decay: 0.0
-  - initial_d:
-      • LoRA: 1e-8
-      • Full fine-tune: 1e-10
-      • Embedding: 1e-7
-  - d_coef: 1
-  - d_limiter: True # To stablizie Prodigy with Simplified_AdEMAMix
-  - factored: False  # Can be true or false, quality should not degrade due to Simplified_AdEMAMix’s high tolerance to 1-bit factorization.
-```
-
-> ✅ **Why it works**:  
-> - `Kourkoutas-β` handles beta2 values
-> - `Simplified_AdEMAMix` ensures responsiveness in small-batch noise
-> - `OrthoGrad` prevents overfitting without weight decay
-
----
-
 ## 📚 References
 
-1. [Revisiting BFloat16 Training](https://arxiv.org/abs/2010.06192)  
-2. [SMMF: Square-Matricized Momentum Factorization](https://arxiv.org/abs/2412.08894)  
-3. [The AdEMAMix Optimizer](https://arxiv.org/abs/2409.03137)  
-4. [Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD](https://arxiv.org/abs/2502.02431)  
-5. [AdaMeM: Memory Efficient Momentum for Adafactor](https://openreview.net/forum?id=fZqMVTz7K5)  
+1. [Revisiting BFloat16 Training](https://arxiv.org/abs/2010.06192)
+2. [SMMF: Square-Matricized Momentum Factorization](https://arxiv.org/abs/2412.08894)
+3. [The AdEMAMix Optimizer](https://arxiv.org/abs/2409.03137)
+4. [Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD](https://arxiv.org/abs/2502.02431)
 6. [Kourkoutas-β: A Sunspike-Driven Adam Optimizer with Desert Flair](https://arxiv.org/abs/2508.12996)
+7. [Scaling Exponents Across Parameterizations and Optimizers](https://arxiv.org/abs/2407.05872)

{adv_optm-1.2.dev9 → adv_optm-2.1.0}/adv_optm/__init__.py

@@ -7,6 +7,7 @@ from .optim import (
     Lion_Prodigy_adv,
     Muon_adv,
     AdaMuon_adv,
+    SignSGD_adv,
 )
 
 __all__ = [
@@ -18,6 +19,7 @@ __all__ = [
     "Lion_Prodigy_adv",
     "Muon_adv",
     "AdaMuon_adv",
+    "SignSGD_adv",
 ]
 
-__version__ = "1.2.dev9"
+__version__ = "2.1.0"