torchzero 0.3.9__tar.gz → 0.3.11__tar.gz

{torchzero-0.3.9 → torchzero-0.3.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: torchzero
-Version: 0.3.9
+Version: 0.3.11
 Summary: Modular optimization library for PyTorch.
 Author-email: Ivan Nikishev <nkshv2@gmail.com>
 License: MIT License
@@ -45,8 +45,6 @@ Dynamic: license-file
 
 `torchzero` is a PyTorch library providing a highly modular framework for creating and experimenting with a huge number of various optimization algorithms - various momentum techniques, gradient clipping, gradient approximations, line searches, quasi newton methods and more. All algorithms are implemented as modules that can be chained together freely.
 
-NOTE: torchzero is in active development, currently docs are in a state of flux.
-
 ## Installation
 
 ```bash
@@ -113,31 +111,21 @@ for epoch in range(100):
 `torchzero` provides a huge number of various modules:
 
 * **Optimizers**: Optimization algorithms.
-  * `Adam`.
-  * `Shampoo`.
-  * `SOAP` (my current recommendation).
-  * `Muon`.
-  * `SophiaH`.
-  * `Adagrad` and `FullMatrixAdagrad`.
-  * `Lion`.
-  * `RMSprop`.
-  * `OrthoGrad`.
-  * `Rprop`.
+  * `Adam`, `Adan`, `Adagrad`, `ESGD`, `FullMatrixAdagrad`, `LMAdagrad`, `AdaHessian`, `AdaptiveHeavyBall`, `OrthoGrad`, `Lion`, `MARS`, `MatrixMomentum`, `AdaptiveMatrixMomentum`, `Muon`, `RMSprop`, `Rprop`, `SAM`, `ASAM`, `MSAM`, `Shampoo`, `SOAP`, `SophiaH`.
 
   Additionally many other optimizers can be easily defined via modules:
   * Grams: `[tz.m.Adam(), tz.m.GradSign()]`
   * LaProp: `[tz.m.RMSprop(), tz.m.EMA(0.9)]`
   * Signum: `[tz.m.HeavyBall(), tz.m.Sign()]`
-  * Full matrix version of any diagonal optimizer, like Adam: `tz.m.FullMatrixAdagrad(beta=0.999, inner=tz.m.EMA(0.9))`
+  * Efficient full-matrix version of any diagonal optimizer, like Adam: `[tz.m.LMAdagrad(beta=0.999, inner=tz.m.EMA(0.9)), tz.m.Debias(0.9, 0.999)]`
   * Cautious version of any optimizer, like SOAP: `[tz.m.SOAP(), tz.m.Cautious()]`
 
 * **Momentum**:
-  * `NAG`: Nesterov Accelerated Gradient.
   * `HeavyBall`: Classic momentum (Polyak's momentum).
+  * `NAG`: Nesterov Accelerated Gradient.
   * `EMA`: Exponential moving average.
-  * `Averaging` (`Medianveraging`, `WeightedAveraging`): Simple, median, or weighted averaging of updates.
+  * `Averaging` (`MedianAveraging`, `WeightedAveraging`): Simple, median, or weighted averaging of updates.
   * `Cautious`, `ScaleByGradCosineSimilarity`: Momentum cautioning.
-  * `MatrixMomentum`, `AdaptiveMatrixMomentum`: Second order momentum.
 
 * **Stabilization**: Gradient stabilization techniques.
   * `ClipNorm`: Clips gradient L2 norm.
@@ -154,31 +142,42 @@ for epoch in range(100):
 
 * **Second order**: Second order methods.
   * `Newton`: Classic Newton's method.
-  * `NewtonCG`: Matrix-free newton's method with conjugate gradient solver.
+  * `InverseFreeNewton`: Inverse-free version of Newton's method.
+  * `NewtonCG`: Matrix-free newton's method with conjugate gradient or minimal residual solvers.
+  * `TruncatedNewtonCG`: Steihaug-Toint Trust-region NewtonCG via a truncated CG solver.
   * `NystromSketchAndSolve`: Nyström sketch-and-solve method.
-  * `NystromPCG`: NewtonCG with Nyström preconditioning (usually beats NewtonCG).
+  * `NystromPCG`: NewtonCG with Nyström preconditioning.
+  * `HigherOrderNewton`: Higher order Newton's method with trust region.
 
 * **Quasi-Newton**: Approximate second-order optimization methods.
   * `LBFGS`: Limited-memory BFGS.
   * `LSR1`: Limited-memory SR1.
   * `OnlineLBFGS`: Online LBFGS.
-  * `BFGS`, `SR1`, `DFP`, `BroydenGood`, `BroydenBad`, `Greenstadt1`, `Greenstadt2`, `ColumnUpdatingMethod`, `ThomasOptimalMethod`, `PSB`, `Pearson2`, `SSVM`: Classic full-matrix quasi-newton methods.
-  * `PolakRibiere`, `FletcherReeves`, `HestenesStiefel`, `DaiYuan`, `LiuStorey`, `ConjugateDescent`, `HagerZhang`, `HybridHS_DY`: Conjugate gradient methods.
+  * `BFGS`, `DFP`, `ICUM`, `PSB`, `SR1`, `SSVM`, `BroydenBad`, `BroydenGood`, `FletcherVMM`, `GradientCorrection`, `Greenstadt1`, `Greenstadt2`, `Horisho`, `McCormick`, `NewSSM`, `Pearson`, `ProjectedNewtonRaphson`, `ThomasOptimalMethod`, `ShorR`: Full-matrix quasi-newton methods.
+  * `DiagonalBFGS`, `DiagonalSR1`, `DiagonalQuasiCauchi`, `DiagonalWeightedQuasiCauchi`, `DNRTR`, `NewDQN`: Diagonal quasi-newton methods.
+  * `PolakRibiere`, `FletcherReeves`, `HestenesStiefel`, `DaiYuan`, `LiuStorey`, `ConjugateDescent`, `HagerZhang`, `HybridHS_DY`, `ProjectedGradientMethod`: Conjugate gradient methods.
+
+* **Trust Region** Trust region can work with exact hessian or any of the quasi-newton methods (L-BFGS support is WIP)
+  * `TrustCG`: Trust-region, uses a Steihaug-Toint truncated CG solver.
+  * `CubicRegularization`: Cubic regularization, works better with exact hessian.
 
 * **Line Search**:
   * `Backtracking`, `AdaptiveBacktracking`: Backtracking line searches (adaptive is my own).
   * `StrongWolfe`: Cubic interpolation line search satisfying strong Wolfe conditions.
   * `ScipyMinimizeScalar`: Wrapper for SciPy's scalar minimization for line search.
-  * `TrustRegion`: First order trust region method.
 
 * **Learning Rate**:
   * `LR`: Controls learning rate and adds support for LR schedulers.
-  * `PolyakStepSize`: Polyak's method.
-  * `Warmup`: Learning rate warmup.
+  * `PolyakStepSize`: Polyak's subgradient method.
+  * `BarzilaiBorwein`: Barzilai-Borwein step-size.
+  * `Warmup`, `WarmupNormCLip`: Learning rate warmup.
 
 * **Projections**: This can implement things like GaLore but I haven't done that yet.
-  * `FFTProjection`, `DCTProjection`: Use any update rule in Fourier or DCT domain (doesn't seem to help though).
-  * `VectorProjection`, `TensorizeProjection`, `BlockPartition`, `TensorNormsProjection`: Structural projection methods (for block BFGS etc.).
+  <!-- * `FFTProjection`, `DCTProjection`: Use any update rule in Fourier or DCT domain (doesn't seem to help though).
+  * `VectorProjection`, `TensorizeProjection`, `BlockPartition`, `TensorNormsProjection`: Structural projection methods (for block BFGS etc.). -->
+  This is WIP
+  * `To`: this casts everything to any other dtype and device for other modules, e.g. if you want better precision
+  * `ViewAsReal`: put if you have complex paramters.
 
 * **Smoothing**: Smoothing-based optimization methods.
   * `LaplacianSmoothing`: Laplacian smoothing for gradients (implements Laplacian Smooth GD).
@@ -194,6 +193,8 @@ for epoch in range(100):
 
 * **Experimental**: various horrible atrocities
 
+A complete list of modules is available in the [documentation](https://torchzero.readthedocs.io/en/latest/autoapi/torchzero/modules/index.html).
+
 ## Advanced Usage
 
 ### Closure
@@ -312,20 +313,21 @@ not in the module itself. Also both per-parameter settings and state are stored
 
 ```python
 import torch
-from torchzero.core import Module, Vars
+from torchzero.core import Module, Var
 
 class HeavyBall(Module):
     def __init__(self, momentum: float = 0.9, dampening: float = 0):
         defaults = dict(momentum=momentum, dampening=dampening)
         super().__init__(defaults)
 
-    def step(self, vars: Vars):
-        # a module takes a Vars object, modifies it or creates a new one, and returns it
-        # Vars has a bunch of attributes, including parameters, gradients, update, closure, loss
+    def step(self, var: Var):
+        # Var object holds all attributes used for optimization - parameters, gradient, update, etc.
+        # a module takes a Var object, modifies it or creates a new one, and returns it
+        # Var has a bunch of attributes, including parameters, gradients, update, closure, loss
         # for now we are only interested in update, and we will apply the heavyball rule to it.
 
-        params = vars.params
-        update = vars.get_update() # list of tensors
+        params = var.params
+        update = var.get_update() # list of tensors
 
         exp_avg_list = []
         for p, u in zip(params, update):
@@ -346,34 +348,57 @@ class HeavyBall(Module):
             # and it is part of self.state
             exp_avg_list.append(buf.clone())
 
-        # set new update to vars
-        vars.update = exp_avg_list
-        return vars
+        # set new update to var
+        var.update = exp_avg_list
+        return var
 ```
 
-There are a some specialized base modules that make it much easier to implement some specific things.
+More in-depth guide will be available in the documentation in the future.
+
+## Other stuff
 
-* `GradApproximator` for gradient approximations
-* `LineSearch` for line searches
-* `Preconditioner` for preconditioners
-* `Projection` for projections like GaLore or into fourier domain.
-* `QuasiNewtonH` for full-matrix quasi-newton methods that update hessian inverse approximation (because they are all very similar)
-* `ConguateGradientBase` for conjugate gradient methods, basically the only difference is how beta is calculated.
+There are also wrappers providing `torch.optim.Optimizer` interface for various other libraries. When using those, make sure closure has `backward` argument as described in **Advanced Usage**.
 
-The documentation on how to actually use them is to write itself in the near future.
+---
 
-## License
+### Scipy
 
-This project is licensed under the MIT License
+#### torchzero.optim.wrappers.scipy.ScipyMinimize
 
-## Project Links
+A wrapper for `scipy.optimize.minimize` with gradients and hessians supplied by pytorch autograd. Scipy provides implementations of the following methods: `'nelder-mead', 'powell', 'cg', 'bfgs', 'newton-cg', 'l-bfgs-b', 'tnc', 'cobyla', 'cobyqa', 'slsqp', 'trust-constr', 'dogleg', 'trust-ncg', 'trust-exact', 'trust-krylov'`.
 
-TODO (there are docs but from very old version)
+#### torchzero.optim.wrappers.scipy.ScipyDE, ScipyDualAnnealing, ScipySHGO, ScipyDIRECT, ScipyBrute
 
-## Other stuff
+Equivalent wrappers for other derivative free solvers available in `scipy.optimize`
+
+---
+
+### NLOpt
+
+#### torchzero.optim.wrappers.nlopt.NLOptWrapper
 
-There are also wrappers providing `torch.optim.Optimizer` interface for for `scipy.optimize`, NLOpt and Nevergrad.
+A wrapper for [NLOpt](https://github.com/stevengj/nlopt) with gradients supplied by pytorch autograd. NLOpt is another popular library with many gradient based and gradient free [algorithms](https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/)
+
+---
+
+### Nevergrad
+
+#### torchzero.optim.wrappers.nevergrad.NevergradWrapper
+
+A wrapper for [nevergrad](https://facebookresearch.github.io/nevergrad/) which has a huge library of gradient free [algorithms](https://facebookresearch.github.io/nevergrad/optimizers_ref.html#optimizers)
+
+---
+
+### fast-cma-es
+
+#### torchzero.optim.wrappers.fcmaes.FcmaesWrapper
+
+A wrapper for [fast-cma-es](https://github.com/dietmarwo/fast-cma-es), which implements various gradient free algorithms. Notably it includes [BITEOPT](https://github.com/avaneev/biteopt) which seems to have very good performance in benchmarks.
+
+# License
+
+This project is licensed under the MIT License
 
-They are in `torchzero.optim.wrappers.scipy.ScipyMinimize`, `torchzero.optim.wrappers.nlopt.NLOptOptimizer`, and `torchzero.optim.wrappers.nevergrad.NevergradOptimizer`. Make sure closure has `backward` argument as described in **Advanced Usage**.
+# Project Links
 
-Apparently https://github.com/avaneev/biteopt is diabolical so I will add a wrapper for it too very soon.
+The documentation is available at <https://torchzero.readthedocs.io/en/latest/>

{torchzero-0.3.9 → torchzero-0.3.11}/README.md

@@ -6,8 +6,6 @@
 
 `torchzero` is a PyTorch library providing a highly modular framework for creating and experimenting with a huge number of various optimization algorithms - various momentum techniques, gradient clipping, gradient approximations, line searches, quasi newton methods and more. All algorithms are implemented as modules that can be chained together freely.
 
-NOTE: torchzero is in active development, currently docs are in a state of flux.
-
 ## Installation
 
 ```bash
@@ -74,31 +72,21 @@ for epoch in range(100):
 `torchzero` provides a huge number of various modules:
 
 * **Optimizers**: Optimization algorithms.
-  * `Adam`.
-  * `Shampoo`.
-  * `SOAP` (my current recommendation).
-  * `Muon`.
-  * `SophiaH`.
-  * `Adagrad` and `FullMatrixAdagrad`.
-  * `Lion`.
-  * `RMSprop`.
-  * `OrthoGrad`.
-  * `Rprop`.
+  * `Adam`, `Adan`, `Adagrad`, `ESGD`, `FullMatrixAdagrad`, `LMAdagrad`, `AdaHessian`, `AdaptiveHeavyBall`, `OrthoGrad`, `Lion`, `MARS`, `MatrixMomentum`, `AdaptiveMatrixMomentum`, `Muon`, `RMSprop`, `Rprop`, `SAM`, `ASAM`, `MSAM`, `Shampoo`, `SOAP`, `SophiaH`.
 
   Additionally many other optimizers can be easily defined via modules:
   * Grams: `[tz.m.Adam(), tz.m.GradSign()]`
   * LaProp: `[tz.m.RMSprop(), tz.m.EMA(0.9)]`
   * Signum: `[tz.m.HeavyBall(), tz.m.Sign()]`
-  * Full matrix version of any diagonal optimizer, like Adam: `tz.m.FullMatrixAdagrad(beta=0.999, inner=tz.m.EMA(0.9))`
+  * Efficient full-matrix version of any diagonal optimizer, like Adam: `[tz.m.LMAdagrad(beta=0.999, inner=tz.m.EMA(0.9)), tz.m.Debias(0.9, 0.999)]`
   * Cautious version of any optimizer, like SOAP: `[tz.m.SOAP(), tz.m.Cautious()]`
 
 * **Momentum**:
-  * `NAG`: Nesterov Accelerated Gradient.
   * `HeavyBall`: Classic momentum (Polyak's momentum).
+  * `NAG`: Nesterov Accelerated Gradient.
   * `EMA`: Exponential moving average.
-  * `Averaging` (`Medianveraging`, `WeightedAveraging`): Simple, median, or weighted averaging of updates.
+  * `Averaging` (`MedianAveraging`, `WeightedAveraging`): Simple, median, or weighted averaging of updates.
   * `Cautious`, `ScaleByGradCosineSimilarity`: Momentum cautioning.
-  * `MatrixMomentum`, `AdaptiveMatrixMomentum`: Second order momentum.
 
 * **Stabilization**: Gradient stabilization techniques.
   * `ClipNorm`: Clips gradient L2 norm.
@@ -115,31 +103,42 @@ for epoch in range(100):
 
 * **Second order**: Second order methods.
   * `Newton`: Classic Newton's method.
-  * `NewtonCG`: Matrix-free newton's method with conjugate gradient solver.
+  * `InverseFreeNewton`: Inverse-free version of Newton's method.
+  * `NewtonCG`: Matrix-free newton's method with conjugate gradient or minimal residual solvers.
+  * `TruncatedNewtonCG`: Steihaug-Toint Trust-region NewtonCG via a truncated CG solver.
   * `NystromSketchAndSolve`: Nyström sketch-and-solve method.
-  * `NystromPCG`: NewtonCG with Nyström preconditioning (usually beats NewtonCG).
+  * `NystromPCG`: NewtonCG with Nyström preconditioning.
+  * `HigherOrderNewton`: Higher order Newton's method with trust region.
 
 * **Quasi-Newton**: Approximate second-order optimization methods.
   * `LBFGS`: Limited-memory BFGS.
   * `LSR1`: Limited-memory SR1.
   * `OnlineLBFGS`: Online LBFGS.
-  * `BFGS`, `SR1`, `DFP`, `BroydenGood`, `BroydenBad`, `Greenstadt1`, `Greenstadt2`, `ColumnUpdatingMethod`, `ThomasOptimalMethod`, `PSB`, `Pearson2`, `SSVM`: Classic full-matrix quasi-newton methods.
-  * `PolakRibiere`, `FletcherReeves`, `HestenesStiefel`, `DaiYuan`, `LiuStorey`, `ConjugateDescent`, `HagerZhang`, `HybridHS_DY`: Conjugate gradient methods.
+  * `BFGS`, `DFP`, `ICUM`, `PSB`, `SR1`, `SSVM`, `BroydenBad`, `BroydenGood`, `FletcherVMM`, `GradientCorrection`, `Greenstadt1`, `Greenstadt2`, `Horisho`, `McCormick`, `NewSSM`, `Pearson`, `ProjectedNewtonRaphson`, `ThomasOptimalMethod`, `ShorR`: Full-matrix quasi-newton methods.
+  * `DiagonalBFGS`, `DiagonalSR1`, `DiagonalQuasiCauchi`, `DiagonalWeightedQuasiCauchi`, `DNRTR`, `NewDQN`: Diagonal quasi-newton methods.
+  * `PolakRibiere`, `FletcherReeves`, `HestenesStiefel`, `DaiYuan`, `LiuStorey`, `ConjugateDescent`, `HagerZhang`, `HybridHS_DY`, `ProjectedGradientMethod`: Conjugate gradient methods.
+
+* **Trust Region** Trust region can work with exact hessian or any of the quasi-newton methods (L-BFGS support is WIP)
+  * `TrustCG`: Trust-region, uses a Steihaug-Toint truncated CG solver.
+  * `CubicRegularization`: Cubic regularization, works better with exact hessian.
 
 * **Line Search**:
   * `Backtracking`, `AdaptiveBacktracking`: Backtracking line searches (adaptive is my own).
   * `StrongWolfe`: Cubic interpolation line search satisfying strong Wolfe conditions.
   * `ScipyMinimizeScalar`: Wrapper for SciPy's scalar minimization for line search.
-  * `TrustRegion`: First order trust region method.
 
 * **Learning Rate**:
   * `LR`: Controls learning rate and adds support for LR schedulers.
-  * `PolyakStepSize`: Polyak's method.
-  * `Warmup`: Learning rate warmup.
+  * `PolyakStepSize`: Polyak's subgradient method.
+  * `BarzilaiBorwein`: Barzilai-Borwein step-size.
+  * `Warmup`, `WarmupNormCLip`: Learning rate warmup.
 
 * **Projections**: This can implement things like GaLore but I haven't done that yet.
-  * `FFTProjection`, `DCTProjection`: Use any update rule in Fourier or DCT domain (doesn't seem to help though).
-  * `VectorProjection`, `TensorizeProjection`, `BlockPartition`, `TensorNormsProjection`: Structural projection methods (for block BFGS etc.).
+  <!-- * `FFTProjection`, `DCTProjection`: Use any update rule in Fourier or DCT domain (doesn't seem to help though).
+  * `VectorProjection`, `TensorizeProjection`, `BlockPartition`, `TensorNormsProjection`: Structural projection methods (for block BFGS etc.). -->
+  This is WIP
+  * `To`: this casts everything to any other dtype and device for other modules, e.g. if you want better precision
+  * `ViewAsReal`: put if you have complex paramters.
 
 * **Smoothing**: Smoothing-based optimization methods.
   * `LaplacianSmoothing`: Laplacian smoothing for gradients (implements Laplacian Smooth GD).
@@ -155,6 +154,8 @@ for epoch in range(100):
 
 * **Experimental**: various horrible atrocities
 
+A complete list of modules is available in the [documentation](https://torchzero.readthedocs.io/en/latest/autoapi/torchzero/modules/index.html).
+
 ## Advanced Usage
 
 ### Closure
@@ -273,20 +274,21 @@ not in the module itself. Also both per-parameter settings and state are stored
 
 ```python
 import torch
-from torchzero.core import Module, Vars
+from torchzero.core import Module, Var
 
 class HeavyBall(Module):
     def __init__(self, momentum: float = 0.9, dampening: float = 0):
         defaults = dict(momentum=momentum, dampening=dampening)
         super().__init__(defaults)
 
-    def step(self, vars: Vars):
-        # a module takes a Vars object, modifies it or creates a new one, and returns it
-        # Vars has a bunch of attributes, including parameters, gradients, update, closure, loss
+    def step(self, var: Var):
+        # Var object holds all attributes used for optimization - parameters, gradient, update, etc.
+        # a module takes a Var object, modifies it or creates a new one, and returns it
+        # Var has a bunch of attributes, including parameters, gradients, update, closure, loss
         # for now we are only interested in update, and we will apply the heavyball rule to it.
 
-        params = vars.params
-        update = vars.get_update() # list of tensors
+        params = var.params
+        update = var.get_update() # list of tensors
 
         exp_avg_list = []
         for p, u in zip(params, update):
@@ -307,34 +309,57 @@ class HeavyBall(Module):
             # and it is part of self.state
             exp_avg_list.append(buf.clone())
 
-        # set new update to vars
-        vars.update = exp_avg_list
-        return vars
+        # set new update to var
+        var.update = exp_avg_list
+        return var
 ```
 
-There are a some specialized base modules that make it much easier to implement some specific things.
+More in-depth guide will be available in the documentation in the future.
+
+## Other stuff
 
-* `GradApproximator` for gradient approximations
-* `LineSearch` for line searches
-* `Preconditioner` for preconditioners
-* `Projection` for projections like GaLore or into fourier domain.
-* `QuasiNewtonH` for full-matrix quasi-newton methods that update hessian inverse approximation (because they are all very similar)
-* `ConguateGradientBase` for conjugate gradient methods, basically the only difference is how beta is calculated.
+There are also wrappers providing `torch.optim.Optimizer` interface for various other libraries. When using those, make sure closure has `backward` argument as described in **Advanced Usage**.
 
-The documentation on how to actually use them is to write itself in the near future.
+---
 
-## License
+### Scipy
 
-This project is licensed under the MIT License
+#### torchzero.optim.wrappers.scipy.ScipyMinimize
 
-## Project Links
+A wrapper for `scipy.optimize.minimize` with gradients and hessians supplied by pytorch autograd. Scipy provides implementations of the following methods: `'nelder-mead', 'powell', 'cg', 'bfgs', 'newton-cg', 'l-bfgs-b', 'tnc', 'cobyla', 'cobyqa', 'slsqp', 'trust-constr', 'dogleg', 'trust-ncg', 'trust-exact', 'trust-krylov'`.
 
-TODO (there are docs but from very old version)
+#### torchzero.optim.wrappers.scipy.ScipyDE, ScipyDualAnnealing, ScipySHGO, ScipyDIRECT, ScipyBrute
 
-## Other stuff
+Equivalent wrappers for other derivative free solvers available in `scipy.optimize`
+
+---
+
+### NLOpt
+
+#### torchzero.optim.wrappers.nlopt.NLOptWrapper
 
-There are also wrappers providing `torch.optim.Optimizer` interface for for `scipy.optimize`, NLOpt and Nevergrad.
+A wrapper for [NLOpt](https://github.com/stevengj/nlopt) with gradients supplied by pytorch autograd. NLOpt is another popular library with many gradient based and gradient free [algorithms](https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/)
+
+---
+
+### Nevergrad
+
+#### torchzero.optim.wrappers.nevergrad.NevergradWrapper
+
+A wrapper for [nevergrad](https://facebookresearch.github.io/nevergrad/) which has a huge library of gradient free [algorithms](https://facebookresearch.github.io/nevergrad/optimizers_ref.html#optimizers)
+
+---
+
+### fast-cma-es
+
+#### torchzero.optim.wrappers.fcmaes.FcmaesWrapper
+
+A wrapper for [fast-cma-es](https://github.com/dietmarwo/fast-cma-es), which implements various gradient free algorithms. Notably it includes [BITEOPT](https://github.com/avaneev/biteopt) which seems to have very good performance in benchmarks.
+
+# License
+
+This project is licensed under the MIT License
 
-They are in `torchzero.optim.wrappers.scipy.ScipyMinimize`, `torchzero.optim.wrappers.nlopt.NLOptOptimizer`, and `torchzero.optim.wrappers.nevergrad.NevergradOptimizer`. Make sure closure has `backward` argument as described in **Advanced Usage**.
+# Project Links
 
-Apparently https://github.com/avaneev/biteopt is diabolical so I will add a wrapper for it too very soon.
+The documentation is available at <https://torchzero.readthedocs.io/en/latest/>

{torchzero-0.3.9 → torchzero-0.3.11}/docs/source/conf.py

@@ -6,10 +6,10 @@
 # -- Project information -----------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 import sys, os
-#sys.path.insert(0, os.path.abspath('.../src'))  
+#sys.path.insert(0, os.path.abspath('.../src'))
 
 project = 'torchzero'
-copyright = '2024, Ivan Nikishev'
+copyright = '2025, Ivan Nikishev'
 author = 'Ivan Nikishev'
 
 # -- General configuration ---------------------------------------------------
@@ -24,10 +24,12 @@ extensions = [
     'sphinx.ext.githubpages',
     'sphinx.ext.napoleon',
     'autoapi.extension',
+    "myst_nb",
+
     # 'sphinx_rtd_theme',
 ]
 autosummary_generate = True
-autoapi_dirs = ['../../src']
+autoapi_dirs = ['../../torchzero']
 autoapi_type = "python"
 # autoapi_ignore = ["*/tensorlist.py"]
 
@@ -48,7 +50,7 @@ exclude_patterns = []
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
 #html_theme = 'alabaster'
-html_theme = 'furo'
+html_theme = 'sphinx_rtd_theme'
 html_static_path = ['_static']
 
 

torchzero-0.3.11/docs/source/docstring template.py

@@ -0,0 +1,46 @@
+class MyModule:
+    """[One-line summary of the class].
+
+    [A more detailed description of the class, explaining its purpose, how it
+    works, and its typical use cases. You can use multiple paragraphs.]
+
+    .. note::
+        [Optional: Add important notes, warnings, or usage guidelines here.
+        For example, you could mention if a closure is required, discuss
+        stability, or highlight performance characteristics. Use the `.. note::`
+        directive to make it stand out in the documentation.]
+
+    Args:
+        param1 (type, optional):
+            [Description of the first parameter. Use :code:`backticks` for
+            inline code like variable names or specific values like ``"autograd"``.
+            Explain what the parameter does.] Defaults to [value].
+        param2 (type):
+            [Description of a mandatory parameter (no "optional" or "Defaults to").]
+        **kwargs:
+            [If you accept keyword arguments, describe what they are used for.]
+
+    Examples:
+        [A title or short sentence describing the first example]:
+
+        .. code-block:: python
+
+            opt = tz.Modular(
+                model.parameters(),
+                ...
+            )
+
+        [A title or short sentence for a second, different example]:
+
+        .. code-block:: python
+
+            opt = tz.Modular(
+                model.parameters(),
+                ...
+            )
+
+    References:
+        - [Optional: A citation for a relevant paper, book, or algorithm.]
+        - [Optional: A link to a blog post or website with more information.]
+
+    """

{torchzero-0.3.9 → torchzero-0.3.11}/pyproject.toml

@@ -1,5 +1,5 @@
 # NEW VERSION TUTORIAL FOR MYSELF
-# STEP 1 - COMMIT NEW CHANGES BUT DON'T PUSH THEM YET
+# STEP 1 - COMMIT NEW CHANGES AND PUSH THEM
 # STEP 2 - BUMP VERSION AND COMMIT IT (DONT PUSH!!!!)
 # STEP 3 - CREATE TAG WITH THAT VERSION
 # STEP 4 - PUSH (SYNC) CHANGES
@@ -13,7 +13,7 @@ build-backend = "setuptools.build_meta"
 name = "torchzero"
 description = "Modular optimization library for PyTorch."
 
-version = "0.3.9"
+version = "0.3.11"
 dependencies = [
   "torch",
   "numpy",

{torchzero-0.3.9 → torchzero-0.3.11}/tests/test_identical.py

@@ -96,8 +96,7 @@ def _assert_identical_device(opt_fn: Callable, merge: bool, use_closure: bool, s
 
 @pytest.mark.parametrize('amsgrad', [True, False])
 def test_adam(amsgrad):
-    # torch_fn = lambda p: torch.optim.Adam(p, lr=1, amsgrad=amsgrad)
-    # pytorch applies debiasing separately so it is applied before epsilo
+    torch_fn = lambda p: torch.optim.Adam(p, lr=1, amsgrad=amsgrad)
     tz_fn = lambda p: tz.Modular(p, tz.m.Adam(amsgrad=amsgrad))
     tz_fn2 = lambda p: tz.Modular(p, tz.m.Adam(amsgrad=amsgrad), tz.m.LR(1)) # test LR fusing
     tz_fn3 = lambda p: tz.Modular(p, tz.m.Adam(amsgrad=amsgrad), tz.m.LR(1), tz.m.Add(1), tz.m.Sub(1))
@@ -133,7 +132,7 @@ def test_adam(amsgrad):
                 tz.m.Debias2(beta=0.999),
                 tz.m.Add(1e-8)]
         ))
-    tz_fns = (tz_fn, tz_fn2, tz_fn3, tz_fn4, tz_fn5, tz_fn_ops, tz_fn_ops2, tz_fn_ops3, tz_fn_ops4)
+    tz_fns = (torch_fn, tz_fn, tz_fn2, tz_fn3, tz_fn4, tz_fn5, tz_fn_ops, tz_fn_ops2, tz_fn_ops3, tz_fn_ops4)
 
     _assert_identical_opts(tz_fns, merge=True, use_closure=True, device='cpu', steps=10)
     for fn in tz_fns: