torchzero 0.1.7__py3-none-any.whl → 0.3.1__py3-none-any.whl

torchzero-0.1.7.dist-info/METADATA

@@ -1,120 +0,0 @@
-Metadata-Version: 2.2
-Name: torchzero
-Version: 0.1.7
-Summary: Modular optimization library for PyTorch.
-Author-email: Ivan Nikishev <nkshv2@gmail.com>
-License: MIT License
-        
-        Copyright (c) 2024 inikishev
-        
-        Permission is hereby granted, free of charge, to any person obtaining a copy
-        of this software and associated documentation files (the "Software"), to deal
-        in the Software without restriction, including without limitation the rights
-        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-        copies of the Software, and to permit persons to whom the Software is
-        furnished to do so, subject to the following conditions:
-        
-        The above copyright notice and this permission notice shall be included in all
-        copies or substantial portions of the Software.
-        
-        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-        SOFTWARE.
-        
-Project-URL: Homepage, https://github.com/inikishev/torchzero
-Project-URL: Repository, https://github.com/inikishev/torchzero
-Project-URL: Issues, https://github.com/inikishev/torchzero/isses
-Keywords: optimization,optimizers,torch,neural networks,zeroth order,second order
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: torch
-Requires-Dist: numpy
-Requires-Dist: typing_extensions
-
-![example workflow](https://github.com/inikishev/torchzero/actions/workflows/tests.yml/badge.svg)
-
-# torchzero
-
-This is a work-in-progress optimizers library for pytorch with composable zeroth, first, second order and quasi newton methods, gradient approximation, line searches and a whole lot of other stuff.
-
-Most optimizers are modular, meaning you can chain them like this:
-
-```py
-optimizer = torchzero.optim.Modular(model.parameters(), [*list of modules*])`
-```
-
-For example you might use `[ClipNorm(4), LR(1e-3), NesterovMomentum(0.9)]` for standard SGD with gradient clipping and nesterov momentum. Move `ClipNorm` to the end to clip the update instead of the gradients. If you don't have access to gradients, add a `RandomizedFDM()` at the beginning to approximate them via randomized finite differences. Add `Cautious()` to make the optimizer cautious.
-
-Each new module takes previous module update and works on it. That way there is no need to reimplement stuff like laplacian smoothing for all optimizers, and it is easy to experiment with grafting, interpolation between different optimizers, and perhaps some weirder combinations like nested momentum.
-
-# How to use
-
-All modules are defined in `torchzero.modules`. You can generally mix and match them however you want. Some pre-made optimizers are available in `torchzero.optim`.
-
-Some optimizers require closure, which should look like this:
-
-```py
-def closure(backward = True):
-  preds = model(inputs)
-  loss = loss_fn(preds, targets)
-
-  # if you can't call loss.backward(), and instead use gradient-free methods,
-  # they always call closure with backward=False.
-  # so you can remove the part below, but keep the unused backward argument.
-  if backward:
-    optimizer.zero_grad()
-    loss.backward()
-  return loss
-
-optimizer.step(closure)
-```
-
-This closure will also work with all built in pytorch optimizers, including LBFGS, all optimizers in this library, as well as most custom ones.
-
-# Contents
-
-Docs are available at [torchzero.readthedocs.io](https://torchzero.readthedocs.io/en/latest/). A preliminary list of all modules is available here <https://torchzero.readthedocs.io/en/latest/autoapi/torchzero/modules/index.html#classes>. Some of the implemented algorithms:
-
-- SGD/Rprop/RMSProp/AdaGrad/Adam as composable modules. They are also tested to exactly match built in pytorch versions.
-- Cautious Optimizers (<https://huggingface.co/papers/2411.16085>)
-- Optimizer grafting (<https://openreview.net/forum?id=FpKgG31Z_i9>)
-- Laplacian smoothing (<https://arxiv.org/abs/1806.06317>)
-- Polyak momentum, nesterov momentum
-- Gradient norm and value clipping, gradient normalization
-- Gradient centralization (<https://arxiv.org/abs/2004.01461>)
-- Learning rate droput (<https://pubmed.ncbi.nlm.nih.gov/35286266/>).
-- Forward gradient (<https://arxiv.org/abs/2202.08587>)
-- Gradient approximation via finite difference or randomized finite difference, which includes SPSA, RDSA, FDSA and Gaussian smoothing (<https://arxiv.org/abs/2211.13566v3>)
-- Various line searches
-- Exact Newton's method (with Levenberg-Marquardt regularization), newton with hessian approximation via finite difference, subspace finite differences newton.
-- Directional newton via one additional forward pass
-
-All modules should be quite fast, especially on models with many different parameters, due to `_foreach` operations.
-
-I am getting to the point where I can start focusing on good docs and tests. As of now, the code should be considered experimental, untested and subject to change, so feel free but be careful if using this for actual project.
-
-# Wrappers
-
-### scipy.optimize.minimize wrapper
-
-scipy.optimize.minimize wrapper with support for both gradient and hessian via batched autograd
-
-```py
-from torchzero.optim.wrappers.scipy import ScipyMinimize
-opt = ScipyMinimize(model.parameters(), method = 'trust-krylov')
-```
-
-Use as any other optimizer (make sure closure accepts `backward` argument like one from **How to use**). Note that it performs full minimization on each step.
-
-### Nevergrad wrapper
-
-```py
-opt = NevergradOptimizer(bench.parameters(), ng.optimizers.NGOptBase, budget = 1000)
-```
-
-Use as any other optimizer (make sure closure accepts `backward` argument like one from **How to use**).

torchzero-0.1.7.dist-info/RECORD

@@ -1,104 +0,0 @@
-torchzero/__init__.py,sha256=CCIYfhGNYMnRP_cdXL7DgocxkEWYUZYgB3Sf1T5tdYY,203
-torchzero/tensorlist.py,sha256=V9m5zJ44PtiJxTOiM7cADxYaWK-ogv5Tk_KnyQqm1oo,41601
-torchzero/core/__init__.py,sha256=hab7HAep0JIVeJ-EQhcOAB9oKIIo2MuCVn7yS3BFVYA,266
-torchzero/core/module.py,sha256=Dhf8Rn6zKpzzO3rzyxzJnRAe_tOKS35AgH-FQQKR16A,22362
-torchzero/core/tensorlist_optimizer.py,sha256=MenRzyNPQJWx26j4Tj2vbX2pEcRu_26HQPIpUu_nsFc,9957
-torchzero/modules/__init__.py,sha256=5f8kt2mMn1eo9YcjXc3ESW-bqMQIRf646V3zlr8UAO4,571
-torchzero/modules/adaptive/__init__.py,sha256=YBVDXCosr4-C-GFCoreHS3DFyHiYMhCbOWgdhVVaZ_E,161
-torchzero/modules/adaptive/adaptive.py,sha256=Zp6cBbuLIeyEDcZJxgpdXPVnw7sjhkBSe9Pd3HHkHXk,6477
-torchzero/modules/experimental/__init__.py,sha256=dRLPbhTO8efkRQHB-8Z29dKS3BEleCso1dsXxgccFgM,647
-torchzero/modules/experimental/experimental.py,sha256=4w3WyBL43WWwYL1ZdoCAdtBHnJfTrUYe9NBmmqTxYes,9808
-torchzero/modules/experimental/quad_interp.py,sha256=6Nsvpv580KpSujPYHxK-qB5m6b7C_dFdBxCXE49VHAQ,4106
-torchzero/modules/experimental/subspace.py,sha256=6n0INUqya24RR4Onm7bGNAPsdqw4lPYIGqT6OCwXNOU,11766
-torchzero/modules/gradient_approximation/__init__.py,sha256=q8rNkk2PK6Y9zk42Mq8pY2YF6fHt5QuJd7BK-RTFKpg,179
-torchzero/modules/gradient_approximation/_fd_formulas.py,sha256=mXqRwsDYjXi3pnI4mkpwwtJE85omYEvcmtbJAOfpg9o,82
-torchzero/modules/gradient_approximation/base_approximator.py,sha256=VjQ_NNLElDnk4kGPYy8iK9A6f2Y6yo6H1axpXkkAeSw,3632
-torchzero/modules/gradient_approximation/fdm.py,sha256=1FVPe3J3tOBgktCKlcd-wOv3bo_FynuF9U4RKbsz-CY,4976
-torchzero/modules/gradient_approximation/forward_gradient.py,sha256=3CC-tcUIAL0d422WHUQLPx0UMcU5URQ5uYCNbgpi19M,6456
-torchzero/modules/gradient_approximation/newton_fdm.py,sha256=BqkNZp00pUBJLWAXKNTk2eggbE4o6u-eLmvWw55zihg,7100
-torchzero/modules/gradient_approximation/rfdm.py,sha256=2nYl_Pu6Lc3gUjTu0GqTImoMSCX5MPFKmvoagpdrf0w,4966
-torchzero/modules/line_search/__init__.py,sha256=hYysFi5b9a5g0jcRNkgZYGRcZ1V7_JacBVWMR7idP38,1380
-torchzero/modules/line_search/armijo.py,sha256=Iw7bsx2yhDhZ0POvD2lc28tv_b5zphOq3MPed8dAYLk,1955
-torchzero/modules/line_search/base_ls.py,sha256=tPlJxZ_CVGPxVrG_jxud8LRbgceZKMf-ZBbDv0jJX2Q,5319
-torchzero/modules/line_search/directional_newton.py,sha256=PMNjjj6n1GVuMe9fwGGYnxbA9Tw5CQxLXgJNhNiwURs,9148
-torchzero/modules/line_search/grid_ls.py,sha256=tCrDxE1OMIXMisYdyiBF56XBRP16mbSs5SFDRcy1vCw,5603
-torchzero/modules/line_search/scipy_minimize_scalar.py,sha256=STGvMuCPcqxw9iaiD6jLsI6YgxAvEic0tfqDQ90qAiA,2196
-torchzero/modules/meta/__init__.py,sha256=ARVR3Vzvq50n-3uFMNxcGUDd2r4Euamay5UYtpIxXNg,407
-torchzero/modules/meta/alternate.py,sha256=kJB3Y507u_a_dra7N0Wo4uTQhbMSfizG0mmmuISzeH8,2119
-torchzero/modules/meta/grafting.py,sha256=IenyBO0FmJDO6st_mfFAUnu98UlChm4Io4aKARsDI0s,7560
-torchzero/modules/meta/optimizer_wrapper.py,sha256=x1nkKoqGK6-lgiuRYzVFcTNUCYeiwFet0MZoaQ3kv8U,6432
-torchzero/modules/meta/return_overrides.py,sha256=Amp4_yiMtXNM8_JgnAKK71BP-NrqvkYha4wt8mA-Nb4,2023
-torchzero/modules/misc/__init__.py,sha256=P43XRz1nnOuJbpq_bQboLJ7hip80SQmvhua2baPdJ-c,390
-torchzero/modules/misc/accumulate.py,sha256=M66TaL0XBioC-2WkPOj8crKoQrWx-np-Im4I2pzvwHU,1324
-torchzero/modules/misc/basic.py,sha256=EnuS6nZwyRXjWytyQef3f5885NC2b79wiQeZSU-urYA,3475
-torchzero/modules/misc/lr.py,sha256=xG1TCPeZazp1SBkAdwxY3tbt9-Tgufzq0noPlcsZkO4,3523
-torchzero/modules/misc/multistep.py,sha256=Jnbs1upMpvabhSkju10ciOuPzMd0CD5UoK4ueUrh048,1770
-torchzero/modules/misc/on_increase.py,sha256=WT86io0qtTk4UbrNAQlXyPD7ad61EagKqhr2mrrRcYA,1998
-torchzero/modules/momentum/__init__.py,sha256=Cj_3KJ76RLX-WQ7xsOoLY9mucQvnkyudTeVH8fnvdwc,138
-torchzero/modules/momentum/momentum.py,sha256=3zpmhzOI7KLX2nbaWivGEquo9nboEmkIyAO8k8SmWdw,3931
-torchzero/modules/operations/__init__.py,sha256=4SxIQMh-ixEqEDXWdizeGOzfhFw-af4Wz0A-00ypmg0,378
-torchzero/modules/operations/multi.py,sha256=3T0mqrnKrdE1G7wayolCZVjWWjLG7geafu1vbClmz3Y,10273
-torchzero/modules/operations/reduction.py,sha256=mMqxW-PO7T-azTnbx2vFnfOVTHfw7u7BzZveoaaCfxc,4621
-torchzero/modules/operations/singular.py,sha256=J1_0HKPi6Tu0j-25lMP4FPRwgjm8drM5QRcUCmvQNV8,3560
-torchzero/modules/optimizers/__init__.py,sha256=QZu8yvqy7ouY45Xju5Z2oHWJiFa1CslknodhWWRZRms,247
-torchzero/modules/optimizers/adagrad.py,sha256=7SQiQBtkN72nhZ6a22p4KCCwzxUYcvt1qPRgT_y2I7w,2014
-torchzero/modules/optimizers/adam.py,sha256=17HJl5VHtUDHPVSW8nTVb9XrjuXQ1H6lFLAaGUkq028,4714
-torchzero/modules/optimizers/lion.py,sha256=DRm09OpQSs2YjFx21OMA5ADp6dVYLPN6uu61GkxyF6o,904
-torchzero/modules/optimizers/rmsprop.py,sha256=vcPjGh1krMLGX97sNKgKc5rx0-tT_cQznSwgmEtlq0w,2152
-torchzero/modules/optimizers/rprop.py,sha256=OjLmHhhPjfRe0KxdPSKduIQs5atwoCvbNLLoDbFWAfI,3603
-torchzero/modules/optimizers/sgd.py,sha256=t31zfKuV6tJRzIOEtZBX0gH_KGdRi4ddvKhYibK5eCM,2242
-torchzero/modules/orthogonalization/__init__.py,sha256=brvrj57U_1qKKU8AUqbe9lyY9jsfzZvUGnvsU4wjDSQ,151
-torchzero/modules/orthogonalization/newtonschulz.py,sha256=gXSkONee9GC5oTODWde2d2HaxTVWbSZMIG1ZD9p5LRE,7517
-torchzero/modules/orthogonalization/svd.py,sha256=HO72YUP4Jlk-WeCoVpT3kqPLuiUwcApNvsXx3-PMFAI,3651
-torchzero/modules/quasi_newton/__init__.py,sha256=G5lW-q0pI47SJ2AZuY4qkjbqfYzJS0qATDo7V0BGzD4,124
-torchzero/modules/regularization/__init__.py,sha256=FD_KERcYY4bdVR22OuKXUUVt63jyfE9V84evwDC1edo,498
-torchzero/modules/regularization/dropout.py,sha256=Wt0rHQxMt2t28fNaNkfNlXVcbcf3TlqWhKzWi1kz6lU,1003
-torchzero/modules/regularization/noise.py,sha256=LFSG4Ga5W2VuT1jzkApKodMiDsHbi8zYJ9N_ZnsLMoI,3026
-torchzero/modules/regularization/normalization.py,sha256=LxegX6Wf0QYZPSxjtUsx7x6IO9UCGc_SBYRfFp3EJHk,12152
-torchzero/modules/regularization/ortho_grad.py,sha256=TLaF8GIM6_tbHHvUhi9XUKJvTdPgF9RM364HXYmVliw,3126
-torchzero/modules/regularization/weight_decay.py,sha256=AjR1RuUDa2DM1PbHx-wBT41XaaAs2w5PKLjRRpV3JrU,3624
-torchzero/modules/scheduling/__init__.py,sha256=NxR1cpKXtZSbVqPRlzzzgH3_JBMuxQCf3nUhmxBN2Cc,89
-torchzero/modules/scheduling/lr_schedulers.py,sha256=MYcQ13YexvyTSD9AHxdu54DSu_neTxY-BImUwBFc7Iw,4963
-torchzero/modules/scheduling/step_size.py,sha256=NTHOFO34GroEE_O2IWmZprRpD9wiyLu7iNvaAUlfVbc,3664
-torchzero/modules/second_order/__init__.py,sha256=oRyRy8mCjurMINHNdsxjlptLbQNU0VnstkDm1Ccv_80,182
-torchzero/modules/second_order/newton.py,sha256=iVNawkyiz-Yj8_cAKi1BYeffk9P0G9UoddnVTBogR64,6751
-torchzero/modules/smoothing/__init__.py,sha256=sKc9P7QJUNEyuoQXjhMkAELFdqe11tB1HnnKeMI4ciQ,194
-torchzero/modules/smoothing/gaussian_smoothing.py,sha256=kP72r-f0prGEEzgaVeY1RLFLTfx05L_SOtLMCrkXeOs,3827
-torchzero/modules/smoothing/laplacian_smoothing.py,sha256=ehb0lj479RRKZTJ5_9FGDXwidWF5soo9hw8vquqrl5s,5552
-torchzero/modules/weight_averaging/__init__.py,sha256=nJJRs68AV2G4rGwiiHNRfm6XmtM-xUev1pCtzNIVfa8,66
-torchzero/modules/weight_averaging/ema.py,sha256=xdVnKC8PxCQCec4ad3ncvznvVsQM5O6EVfsOKsRr18k,2854
-torchzero/modules/weight_averaging/swa.py,sha256=syio5qq1vf76d5bAJU-I8Zc8U12bR2n8C9hEznCgr7s,6764
-torchzero/optim/__init__.py,sha256=vk6pIYJHWAGYJMdtJ1otsmVph-pdL5HwBg-CTeBCGso,253
-torchzero/optim/modular.py,sha256=tiuKXpLqq6jWwWWrfC_QU9XkErl2Saaw_xGau0frwzQ,6060
-torchzero/optim/experimental/__init__.py,sha256=RqNzJu5mVl3T0u7cf4TBzSiA20M1kxTZVYWjSVhEHuU,585
-torchzero/optim/experimental/experimental.py,sha256=tMHZVbEXm3s6mMr7unFSvk_Jks3uAaAG0fzsH6gr098,10928
-torchzero/optim/experimental/ray_search.py,sha256=GYyssL64D6RiImrZ2tchoZJ04x9rX-Bp1y2nQXEGxX0,2662
-torchzero/optim/first_order/__init__.py,sha256=CRT4farcwi8sO1qqDGxXv1856zOwuKlJKBIAIvpL2Z0,336
-torchzero/optim/first_order/cautious.py,sha256=XBeqrLQ4gFKVUYnJI5ROmF9wQJGhY90HR6UG5IS7vYk,6610
-torchzero/optim/first_order/forward_gradient.py,sha256=8u_ya0NuFSopPPhOiqqSov35irg8tF3LeHYg-qthTG0,3065
-torchzero/optim/first_order/optimizers.py,sha256=jYmU6YDsYRGMRsCNkYc6AlvOf3wlU7Uv1xUrzl0o8zo,24501
-torchzero/optim/quasi_newton/__init__.py,sha256=0X83dl-85_j3ck8itWxJR49ZbFeOcWurW6FI8J12F1w,49
-torchzero/optim/quasi_newton/directional_newton.py,sha256=oZ-If8SRcFXTDFKS_zlAcJnif-v5dTCR9HXqmfvsvNA,2595
-torchzero/optim/second_order/__init__.py,sha256=3Gt0dR4NzBK07TV0NF8KZImUGHbI8E2zncDmhIC377I,31
-torchzero/optim/second_order/newton.py,sha256=-DqJrS8JPea6Y9jp5lDV14KyP8A24YKiPxJ32Vsfiv4,3848
-torchzero/optim/wrappers/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-torchzero/optim/wrappers/nevergrad.py,sha256=4PLqfs2L9XJhveyX6l7kJu1cIPl6uv7_UD76amIlP7I,4733
-torchzero/optim/wrappers/nlopt.py,sha256=fGDOZ82sRI2VLH3hKIAhZY4EuKSdu_g217c-NZvD_rs,7104
-torchzero/optim/wrappers/scipy.py,sha256=_BQwFDQ7SBqIA5i1SJ29Xj0jDXVV8MQ_9RcsPT3U6VQ,18047
-torchzero/optim/zeroth_order/__init__.py,sha256=_6T0znO6V63Niq7DMhJPgUuMc_nPvAGxjCjMdf-r64U,218
-torchzero/optim/zeroth_order/fdm.py,sha256=5iJc_F_tRR4cGQfy2Jr8PmAnCGrPva89ZWczSdcBkFk,3686
-torchzero/optim/zeroth_order/newton_fdm.py,sha256=-5E1FGzeJMr8_IougzE_FEOPFt9pEjQxID4Y89Hpmh0,6537
-torchzero/optim/zeroth_order/rfdm.py,sha256=_Y7yiF1bsVRlXt5IK-3zQccwVl95JF0-Xw-fl8Q_7y4,10529
-torchzero/optim/zeroth_order/rs.py,sha256=3w2nnPGWPecourEdUG583vchcqdNxC6Q_PBL3l0PvCk,3333
-torchzero/random/__init__.py,sha256=8EowQhC4yTZuF8w1ZDl73YZtLej8SuhxCk1Bkifbkms,93
-torchzero/random/random.py,sha256=Oq4GvM_6AOsabg5ke6b8h51V9npyHVxp8ob_wC5D-Aw,2829
-torchzero/utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-torchzero/utils/compile.py,sha256=pYEyX8P26iCb_hFqAXC8IP2SSQrRfC7ZDhXS0vVCsfY,1257
-torchzero/utils/derivatives.py,sha256=koLmuUcVcX41SrH_9rvfJyMXyHyocNLuZ-C8Kr2B7hk,4844
-torchzero/utils/python_tools.py,sha256=kkyDhoP695HhapfKrdjcrRbRAbcvB0ArP1pkxuVUlf0,1192
-torchzero/utils/torch_tools.py,sha256=sSBY5Bmk9LOAgPtaq-6TK4wDgPXsg6FIWxv8CVDx82k,3580
-torchzero-0.1.7.dist-info/LICENSE,sha256=r9ZciAoZoqKC_FNADE0ORukj1p1XhLXEbegdsAyqhJs,1087
-torchzero-0.1.7.dist-info/METADATA,sha256=kA1ASlxBXLMOh_T2pgyk2OHsI50sOwguDfDmjg8fgAU,6096
-torchzero-0.1.7.dist-info/WHEEL,sha256=In9FTNxeP60KnTkGw7wk6mJPYd_dQSjEZmXdBdMCI-8,91
-torchzero-0.1.7.dist-info/top_level.txt,sha256=isztuDR1ZGo8p2tORLa-vNuomcbLj7Xd208lhd-pVPs,10
-torchzero-0.1.7.dist-info/RECORD,,

torchzero-0.1.7.dist-info/top_level.txt

@@ -1 +0,0 @@
-torchzero