torchzero 0.1.8__py3-none-any.whl → 0.3.1__py3-none-any.whl

torchzero-0.1.8.dist-info/METADATA

@@ -1,130 +0,0 @@
-Metadata-Version: 2.2
-Name: torchzero
-Version: 0.1.8
-Summary: Modular optimization library for PyTorch.
-Author-email: Ivan Nikishev <nkshv2@gmail.com>
-License: MIT License
-        
-        Copyright (c) 2024 inikishev
-        
-        Permission is hereby granted, free of charge, to any person obtaining a copy
-        of this software and associated documentation files (the "Software"), to deal
-        in the Software without restriction, including without limitation the rights
-        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-        copies of the Software, and to permit persons to whom the Software is
-        furnished to do so, subject to the following conditions:
-        
-        The above copyright notice and this permission notice shall be included in all
-        copies or substantial portions of the Software.
-        
-        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-        SOFTWARE.
-        
-Project-URL: Homepage, https://github.com/inikishev/torchzero
-Project-URL: Repository, https://github.com/inikishev/torchzero
-Project-URL: Issues, https://github.com/inikishev/torchzero/isses
-Keywords: optimization,optimizers,torch,neural networks,zeroth order,second order
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: torch
-Requires-Dist: numpy
-Requires-Dist: typing_extensions
-
-![example workflow](https://github.com/inikishev/torchzero/actions/workflows/tests.yml/badge.svg)
-
-# torchzero
-
-`torchzero` implements a large number of chainable optimization modules that can be chained together to create custom optimizers:
-
-```py
-import torchzero as tz
-
-optimizer = tz.Modular(
-    model.parameters(),
-    tz.m.Adam(),
-    tz.m.Cautious(),
-    tz.m.LR(1e-3),
-    tz.m.WeightDecay(1e-4)
-)
-
-# standard training loop
-for batch in dataset:
-    preds = model(batch)
-    loss = criterion(preds)
-    optimizer.zero_grad()
-    optimizer.step()
-```
-
-Each module takes the output of the previous module and applies a further transformation. This modular design avoids redundant code, such as reimplementing cautioning, orthogonalization, laplacian smoothing, etc for every optimizer. It is also easy to experiment with grafting, interpolation between different optimizers, and perhaps some weirder combinations like nested momentum.
-
-Modules are not limited to gradient transformations. They can perform other operations like line searches, exponential moving average (EMA) and stochastic weight averaging (SWA), gradient accumulation, gradient approximation, and more.
-
-There are over 100 modules, all accessible within the `tz.m` namespace. For example, the Adam update rule is available as `tz.m.Adam`. Complete list of modules is available in [documentation](https://torchzero.readthedocs.io/en/latest/autoapi/torchzero/modules/index.html).
-
-## Closure
-
-Some modules and optimizers in torchzero, particularly line-search methods and gradient approximation modules, require a closure function. This is similar to how `torch.optim.LBFGS` works in PyTorch. In torchzero, closure needs to accept a boolean backward argument (though the argument can have any name). When `backward=True`, the closure should zero out old gradients using `opt.zero_grad()`, and compute new gradients using `loss.backward()`.
-
-```py
-def closure(backward = True):
-    preds = model(inputs)
-    loss = loss_fn(preds, targets)
-
-    if backward:
-        optimizer.zero_grad()
-        loss.backward()
-    return loss
-
-optimizer.step(closure)
-```
-
-If you intend to use gradient-free methods, `backward` argument is still required in the closure. Simply leave it unused. Gradient-free and gradient approximation methods always call closure with `backward=False`.
-
-All built-in pytorch optimizers, as well as most custom ones, support closure too. So the code above will work with all other optimizers out of the box, and you can switch between different optimizers without rewriting your training loop.
-
-# Documentation
-
-For more information on how to create, use and extend torchzero modules, please refer to the documentation at [torchzero.readthedocs.io](https://torchzero.readthedocs.io/en/latest/index.html).
-
-# Extra
-
-Some other optimization related things in torchzero:
-
-### scipy.optimize.minimize wrapper
-
-scipy.optimize.minimize wrapper with support for both gradient and hessian via batched autograd
-
-```py
-from torchzero.optim.wrappers.scipy import ScipyMinimize
-opt = ScipyMinimize(model.parameters(), method = 'trust-krylov')
-```
-
-Use as any other closure-based optimizer, but make sure closure accepts `backward` argument. Note that it performs full minimization on each step.
-
-### Nevergrad wrapper
-
-[Nevergrad](https://github.com/facebookresearch/nevergrad) is an optimization library by facebook with an insane number of gradient free methods.
-
-```py
-from torchzero.optim.wrappers.nevergrad import NevergradOptimizer
-opt = NevergradOptimizer(bench.parameters(), ng.optimizers.NGOptBase, budget = 1000)
-```
-
-Use as any other closure-based optimizer, but make sure closure accepts `backward` argument.
-
-### NLopt wrapper
-
-[NLopt](https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/) is another optimization library similar to scipy.optimize.minimize, with a large number of both gradient based and gradient free methods.
-
-```py
-from torchzero.optim.wrappers.nlopt import NLOptOptimizer
-opt = NLOptOptimizer(bench.parameters(), 'LD_TNEWTON_PRECOND_RESTART', maxeval = 1000)
-```
-
-Use as any other closure-based optimizer, but make sure closure accepts `backward` argument. Note that it performs full minimization on each step.

torchzero-0.1.8.dist-info/RECORD

@@ -1,104 +0,0 @@
-torchzero/__init__.py,sha256=CCIYfhGNYMnRP_cdXL7DgocxkEWYUZYgB3Sf1T5tdYY,203
-torchzero/tensorlist.py,sha256=V9m5zJ44PtiJxTOiM7cADxYaWK-ogv5Tk_KnyQqm1oo,41601
-torchzero/core/__init__.py,sha256=hab7HAep0JIVeJ-EQhcOAB9oKIIo2MuCVn7yS3BFVYA,266
-torchzero/core/module.py,sha256=iFgRX_PY2Y0WD8_w5q6ygry1VhvpsE3jO-ySS_YQkuQ,23023
-torchzero/core/tensorlist_optimizer.py,sha256=MenRzyNPQJWx26j4Tj2vbX2pEcRu_26HQPIpUu_nsFc,9957
-torchzero/modules/__init__.py,sha256=5f8kt2mMn1eo9YcjXc3ESW-bqMQIRf646V3zlr8UAO4,571
-torchzero/modules/adaptive/__init__.py,sha256=YBVDXCosr4-C-GFCoreHS3DFyHiYMhCbOWgdhVVaZ_E,161
-torchzero/modules/adaptive/adaptive.py,sha256=Zp6cBbuLIeyEDcZJxgpdXPVnw7sjhkBSe9Pd3HHkHXk,6477
-torchzero/modules/experimental/__init__.py,sha256=dRLPbhTO8efkRQHB-8Z29dKS3BEleCso1dsXxgccFgM,647
-torchzero/modules/experimental/experimental.py,sha256=4w3WyBL43WWwYL1ZdoCAdtBHnJfTrUYe9NBmmqTxYes,9808
-torchzero/modules/experimental/quad_interp.py,sha256=6Nsvpv580KpSujPYHxK-qB5m6b7C_dFdBxCXE49VHAQ,4106
-torchzero/modules/experimental/subspace.py,sha256=6n0INUqya24RR4Onm7bGNAPsdqw4lPYIGqT6OCwXNOU,11766
-torchzero/modules/gradient_approximation/__init__.py,sha256=q8rNkk2PK6Y9zk42Mq8pY2YF6fHt5QuJd7BK-RTFKpg,179
-torchzero/modules/gradient_approximation/_fd_formulas.py,sha256=mXqRwsDYjXi3pnI4mkpwwtJE85omYEvcmtbJAOfpg9o,82
-torchzero/modules/gradient_approximation/base_approximator.py,sha256=VjQ_NNLElDnk4kGPYy8iK9A6f2Y6yo6H1axpXkkAeSw,3632
-torchzero/modules/gradient_approximation/fdm.py,sha256=1FVPe3J3tOBgktCKlcd-wOv3bo_FynuF9U4RKbsz-CY,4976
-torchzero/modules/gradient_approximation/forward_gradient.py,sha256=3CC-tcUIAL0d422WHUQLPx0UMcU5URQ5uYCNbgpi19M,6456
-torchzero/modules/gradient_approximation/newton_fdm.py,sha256=BqkNZp00pUBJLWAXKNTk2eggbE4o6u-eLmvWw55zihg,7100
-torchzero/modules/gradient_approximation/rfdm.py,sha256=2nYl_Pu6Lc3gUjTu0GqTImoMSCX5MPFKmvoagpdrf0w,4966
-torchzero/modules/line_search/__init__.py,sha256=hYysFi5b9a5g0jcRNkgZYGRcZ1V7_JacBVWMR7idP38,1380
-torchzero/modules/line_search/armijo.py,sha256=Iw7bsx2yhDhZ0POvD2lc28tv_b5zphOq3MPed8dAYLk,1955
-torchzero/modules/line_search/base_ls.py,sha256=tPlJxZ_CVGPxVrG_jxud8LRbgceZKMf-ZBbDv0jJX2Q,5319
-torchzero/modules/line_search/directional_newton.py,sha256=PMNjjj6n1GVuMe9fwGGYnxbA9Tw5CQxLXgJNhNiwURs,9148
-torchzero/modules/line_search/grid_ls.py,sha256=tCrDxE1OMIXMisYdyiBF56XBRP16mbSs5SFDRcy1vCw,5603
-torchzero/modules/line_search/scipy_minimize_scalar.py,sha256=STGvMuCPcqxw9iaiD6jLsI6YgxAvEic0tfqDQ90qAiA,2196
-torchzero/modules/meta/__init__.py,sha256=ARVR3Vzvq50n-3uFMNxcGUDd2r4Euamay5UYtpIxXNg,407
-torchzero/modules/meta/alternate.py,sha256=kJB3Y507u_a_dra7N0Wo4uTQhbMSfizG0mmmuISzeH8,2119
-torchzero/modules/meta/grafting.py,sha256=IenyBO0FmJDO6st_mfFAUnu98UlChm4Io4aKARsDI0s,7560
-torchzero/modules/meta/optimizer_wrapper.py,sha256=x1nkKoqGK6-lgiuRYzVFcTNUCYeiwFet0MZoaQ3kv8U,6432
-torchzero/modules/meta/return_overrides.py,sha256=Amp4_yiMtXNM8_JgnAKK71BP-NrqvkYha4wt8mA-Nb4,2023
-torchzero/modules/misc/__init__.py,sha256=P43XRz1nnOuJbpq_bQboLJ7hip80SQmvhua2baPdJ-c,390
-torchzero/modules/misc/accumulate.py,sha256=M66TaL0XBioC-2WkPOj8crKoQrWx-np-Im4I2pzvwHU,1324
-torchzero/modules/misc/basic.py,sha256=EnuS6nZwyRXjWytyQef3f5885NC2b79wiQeZSU-urYA,3475
-torchzero/modules/misc/lr.py,sha256=xG1TCPeZazp1SBkAdwxY3tbt9-Tgufzq0noPlcsZkO4,3523
-torchzero/modules/misc/multistep.py,sha256=Jnbs1upMpvabhSkju10ciOuPzMd0CD5UoK4ueUrh048,1770
-torchzero/modules/misc/on_increase.py,sha256=WT86io0qtTk4UbrNAQlXyPD7ad61EagKqhr2mrrRcYA,1998
-torchzero/modules/momentum/__init__.py,sha256=Cj_3KJ76RLX-WQ7xsOoLY9mucQvnkyudTeVH8fnvdwc,138
-torchzero/modules/momentum/momentum.py,sha256=3zpmhzOI7KLX2nbaWivGEquo9nboEmkIyAO8k8SmWdw,3931
-torchzero/modules/operations/__init__.py,sha256=4SxIQMh-ixEqEDXWdizeGOzfhFw-af4Wz0A-00ypmg0,378
-torchzero/modules/operations/multi.py,sha256=3T0mqrnKrdE1G7wayolCZVjWWjLG7geafu1vbClmz3Y,10273
-torchzero/modules/operations/reduction.py,sha256=mMqxW-PO7T-azTnbx2vFnfOVTHfw7u7BzZveoaaCfxc,4621
-torchzero/modules/operations/singular.py,sha256=J1_0HKPi6Tu0j-25lMP4FPRwgjm8drM5QRcUCmvQNV8,3560
-torchzero/modules/optimizers/__init__.py,sha256=QZu8yvqy7ouY45Xju5Z2oHWJiFa1CslknodhWWRZRms,247
-torchzero/modules/optimizers/adagrad.py,sha256=7SQiQBtkN72nhZ6a22p4KCCwzxUYcvt1qPRgT_y2I7w,2014
-torchzero/modules/optimizers/adam.py,sha256=17HJl5VHtUDHPVSW8nTVb9XrjuXQ1H6lFLAaGUkq028,4714
-torchzero/modules/optimizers/lion.py,sha256=DRm09OpQSs2YjFx21OMA5ADp6dVYLPN6uu61GkxyF6o,904
-torchzero/modules/optimizers/rmsprop.py,sha256=vcPjGh1krMLGX97sNKgKc5rx0-tT_cQznSwgmEtlq0w,2152
-torchzero/modules/optimizers/rprop.py,sha256=OjLmHhhPjfRe0KxdPSKduIQs5atwoCvbNLLoDbFWAfI,3603
-torchzero/modules/optimizers/sgd.py,sha256=t31zfKuV6tJRzIOEtZBX0gH_KGdRi4ddvKhYibK5eCM,2242
-torchzero/modules/orthogonalization/__init__.py,sha256=brvrj57U_1qKKU8AUqbe9lyY9jsfzZvUGnvsU4wjDSQ,151
-torchzero/modules/orthogonalization/newtonschulz.py,sha256=gXSkONee9GC5oTODWde2d2HaxTVWbSZMIG1ZD9p5LRE,7517
-torchzero/modules/orthogonalization/svd.py,sha256=HO72YUP4Jlk-WeCoVpT3kqPLuiUwcApNvsXx3-PMFAI,3651
-torchzero/modules/quasi_newton/__init__.py,sha256=G5lW-q0pI47SJ2AZuY4qkjbqfYzJS0qATDo7V0BGzD4,124
-torchzero/modules/regularization/__init__.py,sha256=FD_KERcYY4bdVR22OuKXUUVt63jyfE9V84evwDC1edo,498
-torchzero/modules/regularization/dropout.py,sha256=Wt0rHQxMt2t28fNaNkfNlXVcbcf3TlqWhKzWi1kz6lU,1003
-torchzero/modules/regularization/noise.py,sha256=LFSG4Ga5W2VuT1jzkApKodMiDsHbi8zYJ9N_ZnsLMoI,3026
-torchzero/modules/regularization/normalization.py,sha256=LxegX6Wf0QYZPSxjtUsx7x6IO9UCGc_SBYRfFp3EJHk,12152
-torchzero/modules/regularization/ortho_grad.py,sha256=TLaF8GIM6_tbHHvUhi9XUKJvTdPgF9RM364HXYmVliw,3126
-torchzero/modules/regularization/weight_decay.py,sha256=AjR1RuUDa2DM1PbHx-wBT41XaaAs2w5PKLjRRpV3JrU,3624
-torchzero/modules/scheduling/__init__.py,sha256=NxR1cpKXtZSbVqPRlzzzgH3_JBMuxQCf3nUhmxBN2Cc,89
-torchzero/modules/scheduling/lr_schedulers.py,sha256=MYcQ13YexvyTSD9AHxdu54DSu_neTxY-BImUwBFc7Iw,4963
-torchzero/modules/scheduling/step_size.py,sha256=NTHOFO34GroEE_O2IWmZprRpD9wiyLu7iNvaAUlfVbc,3664
-torchzero/modules/second_order/__init__.py,sha256=oRyRy8mCjurMINHNdsxjlptLbQNU0VnstkDm1Ccv_80,182
-torchzero/modules/second_order/newton.py,sha256=iVNawkyiz-Yj8_cAKi1BYeffk9P0G9UoddnVTBogR64,6751
-torchzero/modules/smoothing/__init__.py,sha256=sKc9P7QJUNEyuoQXjhMkAELFdqe11tB1HnnKeMI4ciQ,194
-torchzero/modules/smoothing/gaussian_smoothing.py,sha256=kP72r-f0prGEEzgaVeY1RLFLTfx05L_SOtLMCrkXeOs,3827
-torchzero/modules/smoothing/laplacian_smoothing.py,sha256=ehb0lj479RRKZTJ5_9FGDXwidWF5soo9hw8vquqrl5s,5552
-torchzero/modules/weight_averaging/__init__.py,sha256=nJJRs68AV2G4rGwiiHNRfm6XmtM-xUev1pCtzNIVfa8,66
-torchzero/modules/weight_averaging/ema.py,sha256=xdVnKC8PxCQCec4ad3ncvznvVsQM5O6EVfsOKsRr18k,2854
-torchzero/modules/weight_averaging/swa.py,sha256=syio5qq1vf76d5bAJU-I8Zc8U12bR2n8C9hEznCgr7s,6764
-torchzero/optim/__init__.py,sha256=vk6pIYJHWAGYJMdtJ1otsmVph-pdL5HwBg-CTeBCGso,253
-torchzero/optim/modular.py,sha256=tg-VUcxDAhBzM4uR7SFkf5I2hV_SScfqXCbZpvV1Yzc,6788
-torchzero/optim/experimental/__init__.py,sha256=RqNzJu5mVl3T0u7cf4TBzSiA20M1kxTZVYWjSVhEHuU,585
-torchzero/optim/experimental/experimental.py,sha256=tMHZVbEXm3s6mMr7unFSvk_Jks3uAaAG0fzsH6gr098,10928
-torchzero/optim/experimental/ray_search.py,sha256=GYyssL64D6RiImrZ2tchoZJ04x9rX-Bp1y2nQXEGxX0,2662
-torchzero/optim/first_order/__init__.py,sha256=CRT4farcwi8sO1qqDGxXv1856zOwuKlJKBIAIvpL2Z0,336
-torchzero/optim/first_order/cautious.py,sha256=XBeqrLQ4gFKVUYnJI5ROmF9wQJGhY90HR6UG5IS7vYk,6610
-torchzero/optim/first_order/forward_gradient.py,sha256=8u_ya0NuFSopPPhOiqqSov35irg8tF3LeHYg-qthTG0,3065
-torchzero/optim/first_order/optimizers.py,sha256=jYmU6YDsYRGMRsCNkYc6AlvOf3wlU7Uv1xUrzl0o8zo,24501
-torchzero/optim/quasi_newton/__init__.py,sha256=0X83dl-85_j3ck8itWxJR49ZbFeOcWurW6FI8J12F1w,49
-torchzero/optim/quasi_newton/directional_newton.py,sha256=oZ-If8SRcFXTDFKS_zlAcJnif-v5dTCR9HXqmfvsvNA,2595
-torchzero/optim/second_order/__init__.py,sha256=3Gt0dR4NzBK07TV0NF8KZImUGHbI8E2zncDmhIC377I,31
-torchzero/optim/second_order/newton.py,sha256=-DqJrS8JPea6Y9jp5lDV14KyP8A24YKiPxJ32Vsfiv4,3848
-torchzero/optim/wrappers/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-torchzero/optim/wrappers/nevergrad.py,sha256=4PLqfs2L9XJhveyX6l7kJu1cIPl6uv7_UD76amIlP7I,4733
-torchzero/optim/wrappers/nlopt.py,sha256=fGDOZ82sRI2VLH3hKIAhZY4EuKSdu_g217c-NZvD_rs,7104
-torchzero/optim/wrappers/scipy.py,sha256=_BQwFDQ7SBqIA5i1SJ29Xj0jDXVV8MQ_9RcsPT3U6VQ,18047
-torchzero/optim/zeroth_order/__init__.py,sha256=_6T0znO6V63Niq7DMhJPgUuMc_nPvAGxjCjMdf-r64U,218
-torchzero/optim/zeroth_order/fdm.py,sha256=5iJc_F_tRR4cGQfy2Jr8PmAnCGrPva89ZWczSdcBkFk,3686
-torchzero/optim/zeroth_order/newton_fdm.py,sha256=-5E1FGzeJMr8_IougzE_FEOPFt9pEjQxID4Y89Hpmh0,6537
-torchzero/optim/zeroth_order/rfdm.py,sha256=_Y7yiF1bsVRlXt5IK-3zQccwVl95JF0-Xw-fl8Q_7y4,10529
-torchzero/optim/zeroth_order/rs.py,sha256=3w2nnPGWPecourEdUG583vchcqdNxC6Q_PBL3l0PvCk,3333
-torchzero/random/__init__.py,sha256=8EowQhC4yTZuF8w1ZDl73YZtLej8SuhxCk1Bkifbkms,93
-torchzero/random/random.py,sha256=Oq4GvM_6AOsabg5ke6b8h51V9npyHVxp8ob_wC5D-Aw,2829
-torchzero/utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-torchzero/utils/compile.py,sha256=pYEyX8P26iCb_hFqAXC8IP2SSQrRfC7ZDhXS0vVCsfY,1257
-torchzero/utils/derivatives.py,sha256=koLmuUcVcX41SrH_9rvfJyMXyHyocNLuZ-C8Kr2B7hk,4844
-torchzero/utils/python_tools.py,sha256=kkyDhoP695HhapfKrdjcrRbRAbcvB0ArP1pkxuVUlf0,1192
-torchzero/utils/torch_tools.py,sha256=sSBY5Bmk9LOAgPtaq-6TK4wDgPXsg6FIWxv8CVDx82k,3580
-torchzero-0.1.8.dist-info/LICENSE,sha256=r9ZciAoZoqKC_FNADE0ORukj1p1XhLXEbegdsAyqhJs,1087
-torchzero-0.1.8.dist-info/METADATA,sha256=gtnLFgZ4XQwI7GO5U_p8fUFrt1X8V_Lyv9bvISGnSro,6058
-torchzero-0.1.8.dist-info/WHEEL,sha256=In9FTNxeP60KnTkGw7wk6mJPYd_dQSjEZmXdBdMCI-8,91
-torchzero-0.1.8.dist-info/top_level.txt,sha256=isztuDR1ZGo8p2tORLa-vNuomcbLj7Xd208lhd-pVPs,10
-torchzero-0.1.8.dist-info/RECORD,,

torchzero-0.1.8.dist-info/top_level.txt

@@ -1 +0,0 @@
-torchzero