pytorch-optimizer 3.6.1__tar.gz → 3.7.0__tar.gz

{pytorch_optimizer-3.6.1 → pytorch_optimizer-3.7.0}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.3
 Name: pytorch_optimizer
-Version: 3.6.1
+Version: 3.7.0
 Summary: optimizer & lr scheduler & objective function collections in PyTorch
 License: Apache-2.0
-Keywords: pytorch,deep-learning,optimizer,lr scheduler,A2Grad,Alice,ASGD,AccSGD,AdaBelief,AdaBound,AdaDelta,AdaFactor,AdaGC,AdaMax,AdamG,AdaMod,AdaNorm,AdaPNM,AdaSmooth,AdEMAMix,Simplified-AdEMAMix,ADOPT,AdaHessian,Adai,Adalite,AdaLomo,AdamMini,AdamP,AdamS,Adan,AggMo,Aida,AliG,Amos,Apollo,APOLLO,AvaGrad,bSAM,CAME,DAdaptAdaGrad,DAdaptAdam,DAdaptAdan,DAdaptSGD,DAdaptLion,DeMo,DiffGrad,EXAdam,FAdam,Fira,FOCUS,Fromage,FTRL,GaLore,Grams,Gravity,GrokFast,GSAM,Kate,Lamb,LaProp,LARS,Lion,LOMO,Lookahead,MADGRAD,MARS,MSVAG,Muno,Nero,NovoGrad,OrthoGrad,PAdam,PCGrad,PID,PNM,Prodigy,PSGD,QHAdam,QHM,RACS,RAdam,Ranger,Ranger21,RotoGrad,SAM,GCSAM,LookSAM,ScheduleFreeSGD,ScheduleFreeAdamW,ScheduleFreeRAdam,SCION,SGDP,Shampoo,ScalableShampoo,SGDW,SignSGD,SM3,SOAP,SopihaH,SPAM,StableSPAM,SRMM,StableAdamW,SWATS,TAM,Tiger,TRAC,VSGD,WSAM,Yogi,BCE,BCEFocal,Focal,FocalCosine,SoftF1,Dice,LDAM,Jaccard,Bi-Tempered,Tversky,FocalTversky,LovaszHinge,bitsandbytes,WSD,QGaLore
+Keywords: pytorch,deep-learning,optimizer,lr scheduler,A2Grad,Alice,ASGD,AccSGD,AdaBelief,AdaBound,AdaDelta,AdaFactor,AdaGC,AdaMax,AdaMuon,AdamG,AdaMod,AdaNorm,AdaPNM,AdaSmooth,AdEMAMix,Simplified-AdEMAMix,ADOPT,AdaHessian,Adai,Adalite,AdaLomo,AdamMini,AdamP,AdamS,Adan,AggMo,Aida,AliG,Amos,Apollo,APOLLO,AvaGrad,bSAM,CAME,DAdaptAdaGrad,DAdaptAdam,DAdaptAdan,DAdaptSGD,DAdaptLion,DeMo,DiffGrad,EmoFact,EmoLynx,EmoNavi,EXAdam,FAdam,Fira,FOCUS,Fromage,FTRL,GaLore,Grams,Gravity,GrokFast,GSAM,Kate,Lamb,LaProp,LARS,Lion,LOMO,Lookahead,MADGRAD,MARS,MSVAG,Muno,Nero,NovoGrad,OrthoGrad,PAdam,PCGrad,PID,PNM,Prodigy,PSGD,QHAdam,QHM,RACS,RAdam,Ranger,Ranger21,RotoGrad,SAM,GCSAM,LookSAM,ScheduleFreeSGD,ScheduleFreeAdamW,ScheduleFreeRAdam,SCION,SGDP,Shampoo,ScalableShampoo,SGDW,SignSGD,SM3,SOAP,SopihaH,SPAM,StableSPAM,SPlus,SRMM,StableAdamW,SWATS,TAM,Tiger,TRAC,VSGD,WSAM,Yogi,BCE,BCEFocal,Focal,FocalCosine,SoftF1,Dice,LDAM,Jaccard,Bi-Tempered,Tversky,FocalTversky,LovaszHinge,bitsandbytes,WSD,QGaLore
 Author: kozistr
 Author-email: kozistr@gmail.com
 Maintainer: kozistr
@@ -48,7 +48,7 @@ Description-Content-Type: text/markdown
 
 ## The reasons why you use `pytorch-optimizer`.
 
-* Wide range of supported optimizers. Currently, **108 optimizers (+ `bitsandbytes`, `qgalore`, `torchao`)**, **16 lr schedulers**, and **13 loss functions** are supported!
+* Wide range of supported optimizers. Currently, **110 optimizers (+ `bitsandbytes`, `qgalore`, `torchao`)**, **16 lr schedulers**, and **13 loss functions** are supported!
 * Including many variants such as `ADOPT`, `Cautious`, `AdamD`, `StableAdamW`, and `Gradient Centrailiaztion`
 * Easy to use, clean, and tested codes
 * Active maintenance
@@ -253,7 +253,10 @@ get_supported_optimizers(['adam*', 'ranger*'])
 | RACS & Alice        | *Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension*     |                                                                                                                | <https://arxiv.org/pdf/2502.07752>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250207752G/exportcitation)                                                        |
 | VSGD                | *Variational Stochastic Gradient Descent for Deep Neural Networks*                                             | [github](https://github.com/generativeai-tue/vsgd)                                                             | <https://openreview.net/forum?id=xu4ATNjcdy>                                                | [cite](https://github.com/generativeai-tue/vsgd/tree/main?tab=readme-ov-file#cite)                                                  |
 | SNSM                | *Subset-Norm and Subspace-Momentum: Faster Memory-Efficient Adaptive Optimization with Convergence Guarantees* | [github](https://github.com/timmytonga/sn-sm)                                                                  | <https://arxiv.org/abs/2411.07120>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2024arXiv241107120N/exportcitation)                                                        |
-| AdamC               | Why Gradients Rapidly Increase Near the End of Training*                                                       |                                                                                                                | <https://arxiv.org/abs/2506.02285>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250602285D/exportcitation)                                                        |
+| AdamC               | *Why Gradients Rapidly Increase Near the End of Training*                                                      |                                                                                                                | <https://arxiv.org/abs/2506.02285>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250602285D/exportcitation)                                                        |
+| AdaMuon             | *Adaptive Muon Optimizer*                                                                                      |                                                                                                                | <https://arxiv.org/abs/2507.11005v1>                                                        | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250711005S/exportcitation)                                                        |
+| SPlus               | *A Stable Whitening Optimizer for Efficient Neural Network Training*                                           | [github](https://github.com/kvfrans/splus)                                                                     | <https://arxiv.org/abs/2506.07254>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250607254F/exportcitation)                                                        |
+| EmoNavi             | *An emotion-driven optimizer that feels loss and navigates accordingly.*                                       | [github](https://github.com/muooon/EmoNavi)                                                                    |                                                                                             |                                                                                                                                     |
 
 ## Supported LR Scheduler
 

{pytorch_optimizer-3.6.1 → pytorch_optimizer-3.7.0}/README.md

@@ -10,7 +10,7 @@
 
 ## The reasons why you use `pytorch-optimizer`.
 
-* Wide range of supported optimizers. Currently, **108 optimizers (+ `bitsandbytes`, `qgalore`, `torchao`)**, **16 lr schedulers**, and **13 loss functions** are supported!
+* Wide range of supported optimizers. Currently, **110 optimizers (+ `bitsandbytes`, `qgalore`, `torchao`)**, **16 lr schedulers**, and **13 loss functions** are supported!
 * Including many variants such as `ADOPT`, `Cautious`, `AdamD`, `StableAdamW`, and `Gradient Centrailiaztion`
 * Easy to use, clean, and tested codes
 * Active maintenance
@@ -215,7 +215,10 @@ get_supported_optimizers(['adam*', 'ranger*'])
 | RACS & Alice        | *Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension*     |                                                                                                                | <https://arxiv.org/pdf/2502.07752>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250207752G/exportcitation)                                                        |
 | VSGD                | *Variational Stochastic Gradient Descent for Deep Neural Networks*                                             | [github](https://github.com/generativeai-tue/vsgd)                                                             | <https://openreview.net/forum?id=xu4ATNjcdy>                                                | [cite](https://github.com/generativeai-tue/vsgd/tree/main?tab=readme-ov-file#cite)                                                  |
 | SNSM                | *Subset-Norm and Subspace-Momentum: Faster Memory-Efficient Adaptive Optimization with Convergence Guarantees* | [github](https://github.com/timmytonga/sn-sm)                                                                  | <https://arxiv.org/abs/2411.07120>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2024arXiv241107120N/exportcitation)                                                        |
-| AdamC               | Why Gradients Rapidly Increase Near the End of Training*                                                       |                                                                                                                | <https://arxiv.org/abs/2506.02285>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250602285D/exportcitation)                                                        |
+| AdamC               | *Why Gradients Rapidly Increase Near the End of Training*                                                      |                                                                                                                | <https://arxiv.org/abs/2506.02285>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250602285D/exportcitation)                                                        |
+| AdaMuon             | *Adaptive Muon Optimizer*                                                                                      |                                                                                                                | <https://arxiv.org/abs/2507.11005v1>                                                        | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250711005S/exportcitation)                                                        |
+| SPlus               | *A Stable Whitening Optimizer for Efficient Neural Network Training*                                           | [github](https://github.com/kvfrans/splus)                                                                     | <https://arxiv.org/abs/2506.07254>                                                          | [cite](https://ui.adsabs.harvard.edu/abs/2025arXiv250607254F/exportcitation)                                                        |
+| EmoNavi             | *An emotion-driven optimizer that feels loss and navigates accordingly.*                                       | [github](https://github.com/muooon/EmoNavi)                                                                    |                                                                                             |                                                                                                                                     |
 
 ## Supported LR Scheduler
 

{pytorch_optimizer-3.6.1 → pytorch_optimizer-3.7.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "pytorch_optimizer"
-version = "3.6.1"
+version = "3.7.0"
 description = "optimizer & lr scheduler & objective function collections in PyTorch"
 license = "Apache-2.0"
 authors = ["kozistr <kozistr@gmail.com>"]
@@ -11,18 +11,18 @@ repository = "https://github.com/kozistr/pytorch_optimizer"
 documentation = "https://pytorch-optimizers.readthedocs.io/en/latest"
 keywords = [
     "pytorch", "deep-learning", "optimizer", "lr scheduler", "A2Grad", "Alice", "ASGD", "AccSGD", "AdaBelief",
-    "AdaBound", "AdaDelta", "AdaFactor", "AdaGC", "AdaMax", "AdamG", "AdaMod", "AdaNorm", "AdaPNM", "AdaSmooth",
-    "AdEMAMix", "Simplified-AdEMAMix", "ADOPT", "AdaHessian", "Adai", "Adalite", "AdaLomo", "AdamMini", "AdamP",
-    "AdamS", "Adan", "AggMo", "Aida", "AliG", "Amos", "Apollo", "APOLLO", "AvaGrad", "bSAM", "CAME", "DAdaptAdaGrad",
-    "DAdaptAdam", "DAdaptAdan", "DAdaptSGD", "DAdaptLion", "DeMo", "DiffGrad", "EXAdam", "FAdam", "Fira", "FOCUS",
-    "Fromage", "FTRL", "GaLore", "Grams", "Gravity", "GrokFast", "GSAM", "Kate", "Lamb", "LaProp", "LARS", "Lion",
-    "LOMO", "Lookahead", "MADGRAD", "MARS", "MSVAG", "Muno", "Nero", "NovoGrad", "OrthoGrad", "PAdam", "PCGrad", "PID",
-    "PNM", "Prodigy", "PSGD", "QHAdam", "QHM", "RACS", "RAdam", "Ranger", "Ranger21", "RotoGrad", "SAM", "GCSAM",
-    "LookSAM", "ScheduleFreeSGD", "ScheduleFreeAdamW", "ScheduleFreeRAdam", "SCION", "SGDP", "Shampoo",
-    "ScalableShampoo", "SGDW", "SignSGD", "SM3", "SOAP", "SopihaH", "SPAM", "StableSPAM", "SRMM", "StableAdamW",
-    "SWATS", "TAM", "Tiger", "TRAC", "VSGD", "WSAM", "Yogi", "BCE", "BCEFocal", "Focal", "FocalCosine", "SoftF1",
-    "Dice", "LDAM", "Jaccard", "Bi-Tempered", "Tversky", "FocalTversky", "LovaszHinge", "bitsandbytes", "WSD",
-    "QGaLore",
+    "AdaBound", "AdaDelta", "AdaFactor", "AdaGC", "AdaMax", "AdaMuon", "AdamG", "AdaMod", "AdaNorm", "AdaPNM",
+    "AdaSmooth", "AdEMAMix", "Simplified-AdEMAMix", "ADOPT", "AdaHessian", "Adai", "Adalite", "AdaLomo", "AdamMini",
+    "AdamP", "AdamS", "Adan", "AggMo", "Aida", "AliG", "Amos", "Apollo", "APOLLO", "AvaGrad", "bSAM", "CAME",
+    "DAdaptAdaGrad", "DAdaptAdam", "DAdaptAdan", "DAdaptSGD", "DAdaptLion", "DeMo", "DiffGrad", "EmoFact", "EmoLynx",
+    "EmoNavi", "EXAdam", "FAdam", "Fira", "FOCUS", "Fromage", "FTRL", "GaLore", "Grams", "Gravity", "GrokFast", "GSAM",
+    "Kate", "Lamb", "LaProp", "LARS", "Lion", "LOMO", "Lookahead", "MADGRAD", "MARS", "MSVAG", "Muno", "Nero",
+    "NovoGrad", "OrthoGrad", "PAdam", "PCGrad", "PID", "PNM", "Prodigy", "PSGD", "QHAdam", "QHM", "RACS", "RAdam",
+    "Ranger", "Ranger21", "RotoGrad", "SAM", "GCSAM", "LookSAM", "ScheduleFreeSGD", "ScheduleFreeAdamW",
+    "ScheduleFreeRAdam", "SCION", "SGDP", "Shampoo", "ScalableShampoo", "SGDW", "SignSGD", "SM3", "SOAP", "SopihaH",
+    "SPAM", "StableSPAM", "SPlus", "SRMM", "StableAdamW", "SWATS", "TAM", "Tiger", "TRAC", "VSGD", "WSAM", "Yogi",
+    "BCE", "BCEFocal", "Focal", "FocalCosine", "SoftF1", "Dice", "LDAM", "Jaccard", "Bi-Tempered", "Tversky",
+    "FocalTversky", "LovaszHinge", "bitsandbytes", "WSD", "QGaLore",
 ]
 classifiers = [
     "License :: OSI Approved :: Apache Software License",
@@ -65,9 +65,12 @@ black = [
     { version = "<25", python = "<3.9" },
     { version = "^25", python = ">=3.9" },
 ]
-ruff = "*"
-pytest = "*"
-pytest-cov = "*"
+ruff = "^0.12"
+pytest = "^8"
+pytest-cov = [
+    { version = "<6", python = "<3.9" },
+    { version = "^6", python = ">=3.9" },
+]
 
 [[tool.poetry.source]]
 name = "torch"
@@ -139,5 +142,5 @@ omit = [
 ]
 
 [build-system]
-requires = ["poetry-core>=1.4.0"]
+requires = ["poetry-core>=2.0.0"]
 build-backend = "poetry.core.masonry.api"

{pytorch_optimizer-3.6.1 → pytorch_optimizer-3.7.0}/pytorch_optimizer/__init__.py

@@ -89,6 +89,7 @@ from pytorch_optimizer.optimizer import (
     AdaMod,
     AdamP,
     AdamS,
+    AdaMuon,
     AdamW,
     AdamWSN,
     Adan,
@@ -113,6 +114,9 @@ from pytorch_optimizer.optimizer import (
     DeMo,
     DiffGrad,
     DynamicLossScaler,
+    EmoFact,
+    EmoLynx,
+    EmoNavi,
     EXAdam,
     FAdam,
     Fira,
@@ -153,6 +157,7 @@ from pytorch_optimizer.optimizer import (
     SignSGD,
     SimplifiedAdEMAMix,
     SophiaH,
+    SPlus,
     StableAdamW,
     StableSPAM,
     Tiger,

{pytorch_optimizer-3.6.1 → pytorch_optimizer-3.7.0}/pytorch_optimizer/base/optimizer.py

@@ -154,7 +154,11 @@ class BaseOptimizer(ABC, Optimizer):
 
     @staticmethod
     def apply_ams_bound(
-        ams_bound: bool, exp_avg_sq: torch.Tensor, max_exp_avg_sq: Optional[torch.Tensor], eps: float
+        ams_bound: bool,
+        exp_avg_sq: torch.Tensor,
+        max_exp_avg_sq: Optional[torch.Tensor],
+        eps: float,
+        exp_avg_sq_eps: float = 1e-15,
     ) -> torch.Tensor:
         r"""Apply AMSBound variant.
 
@@ -162,15 +166,16 @@ class BaseOptimizer(ABC, Optimizer):
         :param exp_avg_sq: torch.Tensor. exp_avg_sq.
         :param max_exp_avg_sq: Optional[torch.Tensor]. max_exp_avg_sq.
         :param eps: float. epsilon.
+        :param exp_avg_sq_eps: float. eps value for numerical stability for exp_avg_sq.
         """
         if ams_bound:
             if torch.is_complex(max_exp_avg_sq):
                 max_exp_avg_sq = torch.view_as_real(max_exp_avg_sq)
 
             torch.maximum(max_exp_avg_sq, exp_avg_sq, out=max_exp_avg_sq)
-            de_nom = max_exp_avg_sq.add(eps)
+            de_nom = max_exp_avg_sq.add(exp_avg_sq_eps)
         else:
-            de_nom = exp_avg_sq.add(eps)
+            de_nom = exp_avg_sq.add(exp_avg_sq_eps)
 
         return de_nom.sqrt_().add_(eps)
 

{pytorch_optimizer-3.6.1 → pytorch_optimizer-3.7.0}/pytorch_optimizer/optimizer/__init__.py

@@ -43,6 +43,7 @@ from pytorch_optimizer.optimizer.came import CAME
 from pytorch_optimizer.optimizer.dadapt import DAdaptAdaGrad, DAdaptAdam, DAdaptAdan, DAdaptLion, DAdaptSGD
 from pytorch_optimizer.optimizer.demo import DeMo
 from pytorch_optimizer.optimizer.diffgrad import DiffGrad
+from pytorch_optimizer.optimizer.emonavi import EmoFact, EmoLynx, EmoNavi
 from pytorch_optimizer.optimizer.exadam import EXAdam
 from pytorch_optimizer.optimizer.experimental.ranger25 import Ranger25
 from pytorch_optimizer.optimizer.fadam import FAdam
@@ -66,7 +67,7 @@ from pytorch_optimizer.optimizer.lookahead import Lookahead
 from pytorch_optimizer.optimizer.madgrad import MADGRAD
 from pytorch_optimizer.optimizer.mars import MARS
 from pytorch_optimizer.optimizer.msvag import MSVAG
-from pytorch_optimizer.optimizer.muon import Muon
+from pytorch_optimizer.optimizer.muon import AdaMuon, Muon
 from pytorch_optimizer.optimizer.nero import Nero
 from pytorch_optimizer.optimizer.novograd import NovoGrad
 from pytorch_optimizer.optimizer.orthograd import OrthoGrad
@@ -98,6 +99,7 @@ from pytorch_optimizer.optimizer.snsm import AdamWSN
 from pytorch_optimizer.optimizer.soap import SOAP
 from pytorch_optimizer.optimizer.sophia import SophiaH
 from pytorch_optimizer.optimizer.spam import SPAM, StableSPAM
+from pytorch_optimizer.optimizer.splus import SPlus
 from pytorch_optimizer.optimizer.srmm import SRMM
 from pytorch_optimizer.optimizer.swats import SWATS
 from pytorch_optimizer.optimizer.tam import TAM, AdaTAM
@@ -322,6 +324,11 @@ OPTIMIZER_LIST: List[OPTIMIZER] = [
     RACS,
     Alice,
     VSGD,
+    AdaMuon,
+    SPlus,
+    EmoFact,
+    EmoLynx,
+    EmoNavi,
 ]
 OPTIMIZERS: Dict[str, OPTIMIZER] = {str(optimizer.__name__).lower(): optimizer for optimizer in OPTIMIZER_LIST}
 

pytorch_optimizer-3.7.0/pytorch_optimizer/optimizer/emonavi.py

@@ -0,0 +1,441 @@
+import math
+from typing import Dict, Union
+
+import torch
+
+from pytorch_optimizer.base.exception import NoComplexParameterError, NoSparseGradientError
+from pytorch_optimizer.base.optimizer import BaseOptimizer
+from pytorch_optimizer.base.type import BETAS, CLOSURE, DEFAULTS, GROUP, LOSS, PARAMETERS
+
+
+def update_ema(state: Dict, loss: Union[float, torch.Tensor]) -> Dict[str, float]:
+    r"""Update the EMA dictionary for the `short` and `long` terms."""
+    if isinstance(loss, torch.Tensor):
+        loss = loss.item()
+
+    ema = state.setdefault('ema', {})
+    ema['short'] = 0.3 * loss + 0.7 * ema.get('short', loss)
+    ema['long'] = 0.01 * loss + 0.99 * ema.get('long', loss)
+
+    return ema
+
+
+def compute_scalar(ema: Dict[str, float]) -> float:
+    r"""Compute the difference scalar."""
+    diff: float = ema['short'] - ema['long']
+    return math.tanh(5.0 * diff)
+
+
+def get_scalar_ratio(scalar: float) -> float:
+    r"""Get the scalar ratio."""
+    if scalar > 0.6:
+        return 0.7 + 0.2 * scalar
+    if scalar < -0.6:
+        return 0.1
+    if abs(scalar) > 0.3:
+        return 0.3
+    return 0.0
+
+
+class EmoNavi(BaseOptimizer):
+    r"""An emotion-driven optimizer that feels loss and navigates accordingly.
+
+    :param params: PARAMETERS. iterable of parameters to optimize or dicts defining parameter groups.
+    :param lr: float. learning rate.
+    :param betas: BETAS. coefficients used for computing running averages of gradient and the squared hessian trace.
+    :param shadow_weight: float. the weight of the shadow.
+    :param weight_decay: float. weight decay (L2 penalty).
+    :param weight_decouple: bool. the optimizer uses decoupled weight decay as in AdamW.
+    :param fixed_decay: bool. fix weight decay.
+    :param eps: float. term added to the denominator to improve numerical stability.
+    :param maximize: bool. maximize the objective with respect to the params, instead of minimizing.
+    """
+
+    def __init__(
+        self,
+        params: PARAMETERS,
+        lr: float = 1e-3,
+        betas: BETAS = (0.9, 0.999),
+        shadow_weight: float = 0.05,
+        weight_decay: float = 1e-2,
+        weight_decouple: bool = True,
+        fixed_decay: bool = False,
+        eps: float = 1e-8,
+        maximize: bool = False,
+        **kwargs,
+    ):
+        self.validate_learning_rate(lr)
+        self.validate_betas(betas)
+        self.validate_range(shadow_weight, 'shadow_weight', 0.0, 1.0)
+        self.validate_non_negative(weight_decay, 'weight_decay')
+        self.validate_non_negative(eps, 'eps')
+
+        self.maximize = maximize
+
+        defaults: DEFAULTS = {
+            'lr': lr,
+            'betas': betas,
+            'shadow_weight': shadow_weight,
+            'weight_decay': weight_decay,
+            'weight_decouple': weight_decouple,
+            'fixed_decay': fixed_decay,
+            'eps': eps,
+        }
+
+        super().__init__(params, defaults)
+
+    def __str__(self) -> str:
+        return 'EmoNavi'
+
+    def init_group(self, group: GROUP, **kwargs) -> None:
+        for p in group['params']:
+            if p.grad is None:
+                continue
+
+            grad = p.grad
+            if grad.is_sparse:
+                raise NoSparseGradientError(str(self))
+
+            if torch.is_complex(p):
+                raise NoComplexParameterError(str(self))
+
+            state = self.state[p]
+
+            if len(state) == 0:
+                state['shadow'] = p.clone()
+                state['exp_avg'] = torch.zeros_like(p)
+                state['exp_avg_sq'] = torch.zeros_like(p)
+
+    @torch.no_grad()
+    def step(self, closure: CLOSURE = None) -> LOSS:
+        loss = 0.0
+        if closure is not None:
+            with torch.enable_grad():
+                loss = closure()
+
+        for group in self.param_groups:
+            if 'step' not in group:
+                self.init_group(group)
+                group['step'] = 1
+            else:
+                group['step'] += 1
+
+            beta1, beta2 = group['betas']
+
+            for p in group['params']:
+                if p.grad is None:
+                    continue
+
+                grad = p.grad
+
+                self.maximize_gradient(grad, maximize=self.maximize)
+
+                state = self.state[p]
+
+                ema = update_ema(state, loss)
+                scalar = compute_scalar(ema)
+                ratio = get_scalar_ratio(scalar)
+
+                self.apply_weight_decay(
+                    p=p,
+                    grad=grad,
+                    lr=group['lr'],
+                    weight_decay=group['weight_decay'],
+                    weight_decouple=group['weight_decouple'],
+                    fixed_decay=group['fixed_decay'],
+                )
+
+                shadow, exp_avg, exp_avg_sq = state['shadow'], state['exp_avg'], state['exp_avg_sq']
+
+                if ratio > 0.0:
+                    p.lerp_(shadow, weight=ratio)
+                    shadow.lerp_(p, weight=group['shadow_weight'])
+
+                exp_avg.mul_(beta1).add_(grad, alpha=1.0 - beta1)
+                exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1.0 - beta2)
+
+                de_nom = exp_avg_sq.sqrt().add_(group['eps'])
+
+                p.addcdiv_(exp_avg, de_nom, value=-group['lr'])
+
+        return loss
+
+
+class EmoLynx(BaseOptimizer):
+    r"""EmoLynx optimizer.
+
+    Lynx was developed with inspiration from Lion and Tiger, which we deeply respect for their lightweight and
+    intelligent design. It also integrates EmoNAVI to enhance its capabilities.
+
+    :param params: PARAMETERS. iterable of parameters to optimize or dicts defining parameter groups.
+    :param lr: float. learning rate.
+    :param betas: BETAS. coefficients used for computing running averages of gradient and the squared hessian trace.
+    :param shadow_weight: float. the weight of the shadow.
+    :param weight_decay: float. weight decay (L2 penalty).
+    :param weight_decouple: bool. the optimizer uses decoupled weight decay as in AdamW.
+    :param fixed_decay: bool. fix weight decay.
+    :param eps: float. term added to the denominator to improve numerical stability.
+    :param maximize: bool. maximize the objective with respect to the params, instead of minimizing.
+    """
+
+    def __init__(
+        self,
+        params: PARAMETERS,
+        lr: float = 1e-3,
+        betas: BETAS = (0.9, 0.99),
+        shadow_weight: float = 0.05,
+        weight_decay: float = 1e-2,
+        weight_decouple: bool = True,
+        fixed_decay: bool = False,
+        eps: float = 1e-8,
+        maximize: bool = False,
+        **kwargs,
+    ):
+        self.validate_learning_rate(lr)
+        self.validate_betas(betas)
+        self.validate_range(shadow_weight, 'shadow_weight', 0.0, 1.0)
+        self.validate_non_negative(weight_decay, 'weight_decay')
+        self.validate_non_negative(eps, 'eps')
+
+        self.maximize = maximize
+
+        defaults: DEFAULTS = {
+            'lr': lr,
+            'betas': betas,
+            'shadow_weight': shadow_weight,
+            'weight_decay': weight_decay,
+            'weight_decouple': weight_decouple,
+            'fixed_decay': fixed_decay,
+            'eps': eps,
+        }
+
+        super().__init__(params, defaults)
+
+    def __str__(self) -> str:
+        return 'EmoLynx'
+
+    def init_group(self, group: GROUP, **kwargs) -> None:
+        for p in group['params']:
+            if p.grad is None:
+                continue
+
+            grad = p.grad
+            if grad.is_sparse:
+                raise NoSparseGradientError(str(self))
+
+            if torch.is_complex(p):
+                raise NoComplexParameterError(str(self))
+
+            state = self.state[p]
+
+            if len(state) == 0:
+                state['shadow'] = p.clone()
+                state['exp_avg'] = torch.zeros_like(p)
+
+    @torch.no_grad()
+    def step(self, closure: CLOSURE = None) -> LOSS:
+        loss = 0.0
+        if closure is not None:
+            with torch.enable_grad():
+                loss = closure()
+
+        for group in self.param_groups:
+            if 'step' not in group:
+                self.init_group(group)
+                group['step'] = 1
+            else:
+                group['step'] += 1
+
+            beta1, beta2 = group['betas']
+
+            for p in group['params']:
+                if p.grad is None:
+                    continue
+
+                grad = p.grad
+
+                self.maximize_gradient(grad, maximize=self.maximize)
+
+                state = self.state[p]
+
+                ema = update_ema(state, loss)
+                scalar = compute_scalar(ema)
+                ratio = get_scalar_ratio(scalar)
+
+                self.apply_weight_decay(
+                    p=p,
+                    grad=grad,
+                    lr=group['lr'],
+                    weight_decay=group['weight_decay'],
+                    weight_decouple=group['weight_decouple'],
+                    fixed_decay=group['fixed_decay'],
+                )
+
+                shadow, exp_avg = state['shadow'], state['exp_avg']
+
+                if ratio > 0.0:
+                    p.lerp_(shadow, weight=ratio)
+                    shadow.lerp_(p, weight=group['shadow_weight'])
+
+                blended_grad = grad.mul(1.0 - beta1).add_(exp_avg, alpha=beta1).sign_()
+                exp_avg.mul_(beta2).add_(grad, alpha=1.0 - beta2)
+
+                p.add_(blended_grad, alpha=-group['lr'])
+
+        return loss
+
+
+class EmoFact(BaseOptimizer):
+    r"""EmoFact optimizer.
+
+    EmoFact is inspired by AdaFactor and its VRAM-friendly design is something everyone loves.
+
+    :param params: PARAMETERS. iterable of parameters to optimize or dicts defining parameter groups.
+    :param lr: float. learning rate.
+    :param betas: BETAS. coefficients used for computing running averages of gradient and the squared hessian trace.
+    :param shadow_weight: float. the weight of the shadow.
+    :param weight_decay: float. weight decay (L2 penalty).
+    :param weight_decouple: bool. the optimizer uses decoupled weight decay as in AdamW.
+    :param fixed_decay: bool. fix weight decay.
+    :param eps: float. term added to the denominator to improve numerical stability.
+    :param maximize: bool. maximize the objective with respect to the params, instead of minimizing.
+    """
+
+    def __init__(
+        self,
+        params: PARAMETERS,
+        lr: float = 1e-3,
+        betas: BETAS = (0.9, 0.999),
+        shadow_weight: float = 0.05,
+        weight_decay: float = 1e-2,
+        weight_decouple: bool = True,
+        fixed_decay: bool = False,
+        eps: float = 1e-8,
+        maximize: bool = False,
+        **kwargs,
+    ):
+        self.validate_learning_rate(lr)
+        self.validate_betas(betas)
+        self.validate_range(shadow_weight, 'shadow_weight', 0.0, 1.0)
+        self.validate_non_negative(weight_decay, 'weight_decay')
+        self.validate_non_negative(eps, 'eps')
+
+        self.maximize = maximize
+
+        defaults: DEFAULTS = {
+            'lr': lr,
+            'betas': betas,
+            'shadow_weight': shadow_weight,
+            'weight_decay': weight_decay,
+            'weight_decouple': weight_decouple,
+            'fixed_decay': fixed_decay,
+            'eps': eps,
+        }
+
+        super().__init__(params, defaults)
+
+    def __str__(self) -> str:
+        return 'EmoFact'
+
+    def init_group(self, group: GROUP, **kwargs) -> None:
+        for p in group['params']:
+            if p.grad is None:
+                continue
+
+            grad = p.grad
+            if grad.is_sparse:
+                raise NoSparseGradientError(str(self))
+
+            if torch.is_complex(p):
+                raise NoComplexParameterError(str(self))
+
+            state = self.state[p]
+
+            if len(state) == 0:
+                state['shadow'] = p.clone()
+
+                shape = p.size()
+
+                if len(shape) >= 2:
+                    r_shape = [shape[0]] + [1] * (len(shape) - 1)
+                    state['exp_avg_r'] = torch.zeros(r_shape, dtype=p.dtype, device=p.device)
+
+                    c_shape = [1, *list(shape[1:])]
+                    state['exp_avg_c'] = torch.zeros(c_shape, dtype=p.dtype, device=p.device)
+                else:
+                    state['exp_avg'] = torch.zeros_like(p)
+                    state['exp_avg_sq'] = torch.zeros_like(p)
+
+    @torch.no_grad()
+    def step(self, closure: CLOSURE = None) -> LOSS:
+        loss = 0.0
+        if closure is not None:
+            with torch.enable_grad():
+                loss = closure()
+
+        for group in self.param_groups:
+            if 'step' not in group:
+                self.init_group(group)
+                group['step'] = 1
+            else:
+                group['step'] += 1
+
+            beta1, beta2 = group['betas']
+
+            for p in group['params']:
+                if p.grad is None:
+                    continue
+
+                grad = p.grad
+
+                self.maximize_gradient(grad, maximize=self.maximize)
+
+                state = self.state[p]
+
+                ema = update_ema(state, loss)
+                scalar = compute_scalar(ema)
+                ratio = get_scalar_ratio(scalar)
+
+                self.apply_weight_decay(
+                    p=p,
+                    grad=grad,
+                    lr=group['lr'],
+                    weight_decay=group['weight_decay'],
+                    weight_decouple=group['weight_decouple'],
+                    fixed_decay=group['fixed_decay'],
+                )
+
+                if ratio > 0.0:
+                    shadow = state['shadow']
+
+                    p.lerp_(shadow, weight=ratio)
+                    shadow.lerp_(p, weight=group['shadow_weight'])
+
+                if grad.dim() >= 2:
+                    exp_avg_r, exp_avg_c = state['exp_avg_r'], state['exp_avg_c']
+
+                    grad_p2 = grad.pow(2)
+                    r_sq = (
+                        torch.mean(grad_p2, dim=tuple(range(1, grad.dim())), keepdim=True).add_(group['eps']).sqrt_()
+                    )
+                    c_sq = torch.mean(grad_p2, dim=0, keepdim=True).add_(group['eps']).sqrt_()
+
+                    exp_avg_r.mul_(beta1).add_(r_sq, alpha=1.0 - beta1)
+                    exp_avg_c.mul_(beta1).add_(c_sq, alpha=1.0 - beta1)
+
+                    de_nom = (exp_avg_r * exp_avg_c).sqrt_().add_(group['eps'])
+
+                    update = grad / de_nom
+                else:
+                    exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
+
+                    exp_avg.mul_(beta1).add_(grad, alpha=1.0 - beta1)
+                    exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1.0 - beta2)
+
+                    de_nom = exp_avg_sq.sqrt().add_(group['eps'])
+
+                    update = exp_avg / de_nom
+
+                p.add_(update, alpha=-group['lr'])
+
+        return loss