ista-daslab-optimizers 0.0.1__tar.gz → 1.0.1__tar.gz

{ista_daslab_optimizers-0.0.1/ista_daslab_optimizers.egg-info → ista_daslab_optimizers-1.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ista_daslab_optimizers
-Version: 0.0.1
+Version: 1.0.1
 Summary: Deep Learning optimizers developed in the Distributed Algorithms and Systems group (DASLab) @ Institute of Science and Technology Austria (ISTA)
 Author-email: Ionut-Vlad Modoranu <ionut-vlad.modoranu@ist.ac.at>
 Maintainer-email: Ionut-Vlad Modoranu <ionut-vlad.modoranu@ist.ac.at>
@@ -208,59 +208,66 @@ License:                                  Apache License
         
 Project-URL: Repository, https://github.com/IST-DASLab/ISTA-DASLab-Optimizers
 Keywords: adaptive optimization,deep learning,low memory optimization
-Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.8
 Classifier: License :: OSI Approved :: Apache Software License
-Requires-Python: >=3.9
+Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: torch>=2.3.1
-Requires-Dist: torchaudio>=2.3.1
-Requires-Dist: torchvision>=0.18.1
-Requires-Dist: numpy>=1.24.1
-Requires-Dist: wandb>=0.17.1
-Requires-Dist: gpustat>=1.1.1
-Requires-Dist: timm>=1.0.3
-Requires-Dist: einops>=0.8.0
-Requires-Dist: psutil>=5.9.8
+Requires-Dist: torch
+Requires-Dist: torchaudio
+Requires-Dist: torchvision
+Requires-Dist: numpy
+Requires-Dist: wandb
+Requires-Dist: gpustat
+Requires-Dist: timm
+Requires-Dist: einops
+Requires-Dist: psutil
 
 # ISTA DAS Lab Optimization Algorithms Package
 This repository contains optimization algorithms for Deep Learning developed by 
 the Distributed Algorithms and Systems lab at Institute of Science and Technology Austria.
 
-## Project status
-- **June 5th, 2024**:
-  - *DONE*: the project is locally installable via `pip install .` 
-  - *NEXT*:
-    - working on examples for Sparse M-FAC and Dense M-FAC
-- **May 27th, 2024**:
-  - we are currently working on solving the issues with the installation via `pip`. 
+The repository contains code for the following optimizers published by DASLab @ ISTA:
+- **AC/DC**:
+  - paper: [AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks](https://arxiv.org/abs/2106.12379)
+  - official repository: [GitHub](https://github.com/IST-DASLab/ACDC)
+- **M-FAC**:
+  - paper: [M-FAC: Efficient Matrix-Free Approximations of Second-Order Information](https://arxiv.org/abs/2107.03356)
+  - official repository: [GitHub](https://github.com/IST-DASLab/M-FAC)
+- **Sparse M-FAC with Error Feedback**:
+  - paper: [Error Feedback Can Accurately Compress Preconditioners](https://arxiv.org/abs/2306.06098)
+  - official repository: [GitHub](https://github.com/IST-DASLab/EFCP/)
+- **MicroAdam**:
+  - paper: [MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence](https://arxiv.org/abs/2405.15593)
+  - official repository: [GitHub](https://github.com/IST-DASLab/MicroAdam)
 
 ### Installation
-We provide a script `install.sh` that creates a new environment, installs requirements 
-and then builds the optimizers project. First of all, you have to clone this repository, then 
-run the installation script.
+To use the latest stable version of the repository, you can install via pip:
+
+```shell
+pip3 install ista-daslab-optimizers
+```
+
+We also provide a script `install.sh` that creates a new environment, installs requirements
+and then installs the project as a Python package following these steps:
+
 ```shell
 git clone git@github.com:IST-DASLab/ISTA-DASLab-Optimizers.git
 cd ISTA-DASLab-Optimizers
 source install.sh
 ```
 
-### ⚠️ Important Notice ⚠️
-We noticed it is useful to compile the kernels for each individual CUDA capability separately. For example, for CUDA capability (CC) 8.6, 
-the CUDA kernels for `MicroAdam` will be installed in the package `micro_adam_sm86`, while for CC 9.0 it will be installed in the package
-`micro_adam_sm90`. Please install this library for each system where the CC is different to cover all possible cases for your system. The 
-code will automatically detect the CC version and import the correct package if installed, otherwise will throw an error. The code that 
-dynamically detects the CC version can be found 
-[here](https://github.com/IST-DASLab/ISTA-DASLab-Optimizers/blob/main/ista_daslab_optimizers/tools.py#L17).
-
 ## How to use optimizers?
 
-We provide a minimal working example with ResNet-18 and CIFAR-10 for optimizers `micro-adam`, `acdc`, `sparse-mfac`, `dense-mfac`:
+In this repository we provide a minimal working example for CIFAR-10 for optimizers `acdc`, `dense_mfac`, `sparse_mfac` and `micro_adam`:
 ```shell
-OPTIMIZER=micro-adam # or any other optimizer listed above
+cd examples/cifar10
+OPTIMIZER=micro_adam # or any other optimizer listed above
 bash run_${OPTIMIZER}.sh
 ```
 
+To integrate the optimizers into your own pipeline, you can use the following snippets:
+
 ### MicroAdam optimizer
 ```python
 from ista_daslab_optimizers import MicroAdam
@@ -277,3 +284,19 @@ optimizer = MicroAdam(
 
 # from now on, you can use the variable `optimizer` as any other PyTorch optimizer
 ```
+
+# Versions summary:
+
+---
+
+- **1.0.1** @ June 27th, 2024:
+
+  - removed version in dependencies to avoid conflicts with llm-foundry
+
+- **1.0.0** @ June 20th, 2024:
+  
+  - changed minimum required Python version to 3.8+ and torch to 2.3.0+
+
+- **0.0.1** @ June 13th, 2024:
+
+  - added initial version of the package for Python 3.9+ and torch 2.3.1+

ista_daslab_optimizers-1.0.1/README.md

@@ -0,0 +1,77 @@
+# ISTA DAS Lab Optimization Algorithms Package
+This repository contains optimization algorithms for Deep Learning developed by 
+the Distributed Algorithms and Systems lab at Institute of Science and Technology Austria.
+
+The repository contains code for the following optimizers published by DASLab @ ISTA:
+- **AC/DC**:
+  - paper: [AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks](https://arxiv.org/abs/2106.12379)
+  - official repository: [GitHub](https://github.com/IST-DASLab/ACDC)
+- **M-FAC**:
+  - paper: [M-FAC: Efficient Matrix-Free Approximations of Second-Order Information](https://arxiv.org/abs/2107.03356)
+  - official repository: [GitHub](https://github.com/IST-DASLab/M-FAC)
+- **Sparse M-FAC with Error Feedback**:
+  - paper: [Error Feedback Can Accurately Compress Preconditioners](https://arxiv.org/abs/2306.06098)
+  - official repository: [GitHub](https://github.com/IST-DASLab/EFCP/)
+- **MicroAdam**:
+  - paper: [MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence](https://arxiv.org/abs/2405.15593)
+  - official repository: [GitHub](https://github.com/IST-DASLab/MicroAdam)
+
+### Installation
+To use the latest stable version of the repository, you can install via pip:
+
+```shell
+pip3 install ista-daslab-optimizers
+```
+
+We also provide a script `install.sh` that creates a new environment, installs requirements
+and then installs the project as a Python package following these steps:
+
+```shell
+git clone git@github.com:IST-DASLab/ISTA-DASLab-Optimizers.git
+cd ISTA-DASLab-Optimizers
+source install.sh
+```
+
+## How to use optimizers?
+
+In this repository we provide a minimal working example for CIFAR-10 for optimizers `acdc`, `dense_mfac`, `sparse_mfac` and `micro_adam`:
+```shell
+cd examples/cifar10
+OPTIMIZER=micro_adam # or any other optimizer listed above
+bash run_${OPTIMIZER}.sh
+```
+
+To integrate the optimizers into your own pipeline, you can use the following snippets:
+
+### MicroAdam optimizer
+```python
+from ista_daslab_optimizers import MicroAdam
+
+model = MyCustomModel()
+
+optimizer = MicroAdam(
+    model.parameters(), # or some custom parameter groups
+    m=10, # sliding window size (number of gradients)
+    lr=1e-5, # change accordingly
+    quant_block_size=100_000, # 32 or 64 also works
+    k_init=0.01, # float between 0 and 1 meaning percentage: 0.01 means 1%
+)
+
+# from now on, you can use the variable `optimizer` as any other PyTorch optimizer
+```
+
+# Versions summary:
+
+---
+
+- **1.0.1** @ June 27th, 2024:
+
+  - removed version in dependencies to avoid conflicts with llm-foundry
+
+- **1.0.0** @ June 20th, 2024:
+  
+  - changed minimum required Python version to 3.8+ and torch to 2.3.0+
+
+- **0.0.1** @ June 13th, 2024:
+
+  - added initial version of the package for Python 3.9+ and torch 2.3.1+

{ista_daslab_optimizers-0.0.1 → ista_daslab_optimizers-1.0.1/ista_daslab_optimizers.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ista_daslab_optimizers
-Version: 0.0.1
+Version: 1.0.1
 Summary: Deep Learning optimizers developed in the Distributed Algorithms and Systems group (DASLab) @ Institute of Science and Technology Austria (ISTA)
 Author-email: Ionut-Vlad Modoranu <ionut-vlad.modoranu@ist.ac.at>
 Maintainer-email: Ionut-Vlad Modoranu <ionut-vlad.modoranu@ist.ac.at>
@@ -208,59 +208,66 @@ License:                                  Apache License
         
 Project-URL: Repository, https://github.com/IST-DASLab/ISTA-DASLab-Optimizers
 Keywords: adaptive optimization,deep learning,low memory optimization
-Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.8
 Classifier: License :: OSI Approved :: Apache Software License
-Requires-Python: >=3.9
+Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: torch>=2.3.1
-Requires-Dist: torchaudio>=2.3.1
-Requires-Dist: torchvision>=0.18.1
-Requires-Dist: numpy>=1.24.1
-Requires-Dist: wandb>=0.17.1
-Requires-Dist: gpustat>=1.1.1
-Requires-Dist: timm>=1.0.3
-Requires-Dist: einops>=0.8.0
-Requires-Dist: psutil>=5.9.8
+Requires-Dist: torch
+Requires-Dist: torchaudio
+Requires-Dist: torchvision
+Requires-Dist: numpy
+Requires-Dist: wandb
+Requires-Dist: gpustat
+Requires-Dist: timm
+Requires-Dist: einops
+Requires-Dist: psutil
 
 # ISTA DAS Lab Optimization Algorithms Package
 This repository contains optimization algorithms for Deep Learning developed by 
 the Distributed Algorithms and Systems lab at Institute of Science and Technology Austria.
 
-## Project status
-- **June 5th, 2024**:
-  - *DONE*: the project is locally installable via `pip install .` 
-  - *NEXT*:
-    - working on examples for Sparse M-FAC and Dense M-FAC
-- **May 27th, 2024**:
-  - we are currently working on solving the issues with the installation via `pip`. 
+The repository contains code for the following optimizers published by DASLab @ ISTA:
+- **AC/DC**:
+  - paper: [AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks](https://arxiv.org/abs/2106.12379)
+  - official repository: [GitHub](https://github.com/IST-DASLab/ACDC)
+- **M-FAC**:
+  - paper: [M-FAC: Efficient Matrix-Free Approximations of Second-Order Information](https://arxiv.org/abs/2107.03356)
+  - official repository: [GitHub](https://github.com/IST-DASLab/M-FAC)
+- **Sparse M-FAC with Error Feedback**:
+  - paper: [Error Feedback Can Accurately Compress Preconditioners](https://arxiv.org/abs/2306.06098)
+  - official repository: [GitHub](https://github.com/IST-DASLab/EFCP/)
+- **MicroAdam**:
+  - paper: [MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence](https://arxiv.org/abs/2405.15593)
+  - official repository: [GitHub](https://github.com/IST-DASLab/MicroAdam)
 
 ### Installation
-We provide a script `install.sh` that creates a new environment, installs requirements 
-and then builds the optimizers project. First of all, you have to clone this repository, then 
-run the installation script.
+To use the latest stable version of the repository, you can install via pip:
+
+```shell
+pip3 install ista-daslab-optimizers
+```
+
+We also provide a script `install.sh` that creates a new environment, installs requirements
+and then installs the project as a Python package following these steps:
+
 ```shell
 git clone git@github.com:IST-DASLab/ISTA-DASLab-Optimizers.git
 cd ISTA-DASLab-Optimizers
 source install.sh
 ```
 
-### ⚠️ Important Notice ⚠️
-We noticed it is useful to compile the kernels for each individual CUDA capability separately. For example, for CUDA capability (CC) 8.6, 
-the CUDA kernels for `MicroAdam` will be installed in the package `micro_adam_sm86`, while for CC 9.0 it will be installed in the package
-`micro_adam_sm90`. Please install this library for each system where the CC is different to cover all possible cases for your system. The 
-code will automatically detect the CC version and import the correct package if installed, otherwise will throw an error. The code that 
-dynamically detects the CC version can be found 
-[here](https://github.com/IST-DASLab/ISTA-DASLab-Optimizers/blob/main/ista_daslab_optimizers/tools.py#L17).
-
 ## How to use optimizers?
 
-We provide a minimal working example with ResNet-18 and CIFAR-10 for optimizers `micro-adam`, `acdc`, `sparse-mfac`, `dense-mfac`:
+In this repository we provide a minimal working example for CIFAR-10 for optimizers `acdc`, `dense_mfac`, `sparse_mfac` and `micro_adam`:
 ```shell
-OPTIMIZER=micro-adam # or any other optimizer listed above
+cd examples/cifar10
+OPTIMIZER=micro_adam # or any other optimizer listed above
 bash run_${OPTIMIZER}.sh
 ```
 
+To integrate the optimizers into your own pipeline, you can use the following snippets:
+
 ### MicroAdam optimizer
 ```python
 from ista_daslab_optimizers import MicroAdam
@@ -277,3 +284,19 @@ optimizer = MicroAdam(
 
 # from now on, you can use the variable `optimizer` as any other PyTorch optimizer
 ```
+
+# Versions summary:
+
+---
+
+- **1.0.1** @ June 27th, 2024:
+
+  - removed version in dependencies to avoid conflicts with llm-foundry
+
+- **1.0.0** @ June 20th, 2024:
+  
+  - changed minimum required Python version to 3.8+ and torch to 2.3.0+
+
+- **0.0.1** @ June 13th, 2024:
+
+  - added initial version of the package for Python 3.9+ and torch 2.3.1+

ista_daslab_optimizers-1.0.1/ista_daslab_optimizers.egg-info/requires.txt

@@ -0,0 +1,9 @@
+torch
+torchaudio
+torchvision
+numpy
+wandb
+gpustat
+timm
+einops
+psutil

{ista_daslab_optimizers-0.0.1 → ista_daslab_optimizers-1.0.1}/pyproject.toml

@@ -1,22 +1,22 @@
 [build-system]
-requires = ["setuptools", "wheel", "torch>=2.3.1"]
+requires = ["setuptools", "wheel", "torch"]
 build-backend = "setuptools.build_meta"
 
 [project]
 name='ista_daslab_optimizers'
-version='0.0.1'
+version='1.0.1'
 dependencies = [
-    "torch>=2.3.1",
-    "torchaudio>=2.3.1",
-    "torchvision>=0.18.1",
-    "numpy>=1.24.1",
-    "wandb>=0.17.1",
-    "gpustat>=1.1.1",
-    "timm>=1.0.3",
-    "einops>=0.8.0",
-    "psutil>=5.9.8",
+    "torch", # >=2.3.1",
+    "torchaudio", # >=2.3.1",
+    "torchvision", #>=0.18.1",
+    "numpy", # >=1.24.1",
+    "wandb",#>=0.17.1",
+    "gpustat",#>=1.1.1",
+    "timm", # >=1.0.3",
+    "einops", # >=0.7.0",
+    "psutil", # >=5.9.8",
 ]
-requires-python = '>= 3.9'
+requires-python = '>= 3.8'
 authors = [
     {name = "Ionut-Vlad Modoranu", email = "ionut-vlad.modoranu@ist.ac.at"}
 ]
@@ -32,7 +32,7 @@ keywords = [
     "low memory optimization",
 ]
 classifiers = [
-    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.8",
     "License :: OSI Approved :: Apache Software License",
 ]
 

{ista_daslab_optimizers-0.0.1 → ista_daslab_optimizers-1.0.1}/setup.py

@@ -1,22 +1,19 @@
 from setuptools import setup, find_packages
 from torch.utils.cpp_extension import CUDAExtension, BuildExtension
-import os
 
-# CURRENT_PATH = os.environ['CURRENT_PATH']
-# kernels_dir = './kernels' # os.path.join(CURRENT_PATH, 'kernels')
-# cwd = os.getcwd()
+"""
+    How to add headers when building the project using `python3 -m build` (https://stackoverflow.com/a/6681343/22855002)
+    - Add the relative path to MANIFEST.in file
 
-# print('-' * 100)
-# print(f'{kernels_dir=}')
-# print(f'{cwd=}')
-# print('-' * 100)
+    What didn't work:
+    - headers parameter of setup function
+    - include_dirs parameter of CUDAExtension
+"""
 
 def get_cuda_extension(name, sources):
     return CUDAExtension(
         name=name,
         sources=sources,
-        # include_dirs=['/nfs/scistore19/alistgrp/imodoran/workplace/ISTA-DASLab-Optimizers/kernels'],
-        # library_dirs=[kernels_dir],
     )
 
 setup(

ista_daslab_optimizers-0.0.1/README.md

@@ -1,54 +0,0 @@
-# ISTA DAS Lab Optimization Algorithms Package
-This repository contains optimization algorithms for Deep Learning developed by 
-the Distributed Algorithms and Systems lab at Institute of Science and Technology Austria.
-
-## Project status
-- **June 5th, 2024**:
-  - *DONE*: the project is locally installable via `pip install .` 
-  - *NEXT*:
-    - working on examples for Sparse M-FAC and Dense M-FAC
-- **May 27th, 2024**:
-  - we are currently working on solving the issues with the installation via `pip`. 
-
-### Installation
-We provide a script `install.sh` that creates a new environment, installs requirements 
-and then builds the optimizers project. First of all, you have to clone this repository, then 
-run the installation script.
-```shell
-git clone git@github.com:IST-DASLab/ISTA-DASLab-Optimizers.git
-cd ISTA-DASLab-Optimizers
-source install.sh
-```
-
-### ⚠️ Important Notice ⚠️
-We noticed it is useful to compile the kernels for each individual CUDA capability separately. For example, for CUDA capability (CC) 8.6, 
-the CUDA kernels for `MicroAdam` will be installed in the package `micro_adam_sm86`, while for CC 9.0 it will be installed in the package
-`micro_adam_sm90`. Please install this library for each system where the CC is different to cover all possible cases for your system. The 
-code will automatically detect the CC version and import the correct package if installed, otherwise will throw an error. The code that 
-dynamically detects the CC version can be found 
-[here](https://github.com/IST-DASLab/ISTA-DASLab-Optimizers/blob/main/ista_daslab_optimizers/tools.py#L17).
-
-## How to use optimizers?
-
-We provide a minimal working example with ResNet-18 and CIFAR-10 for optimizers `micro-adam`, `acdc`, `sparse-mfac`, `dense-mfac`:
-```shell
-OPTIMIZER=micro-adam # or any other optimizer listed above
-bash run_${OPTIMIZER}.sh
-```
-
-### MicroAdam optimizer
-```python
-from ista_daslab_optimizers import MicroAdam
-
-model = MyCustomModel()
-
-optimizer = MicroAdam(
-    model.parameters(), # or some custom parameter groups
-    m=10, # sliding window size (number of gradients)
-    lr=1e-5, # change accordingly
-    quant_block_size=100_000, # 32 or 64 also works
-    k_init=0.01, # float between 0 and 1 meaning percentage: 0.01 means 1%
-)
-
-# from now on, you can use the variable `optimizer` as any other PyTorch optimizer
-```

ista_daslab_optimizers-0.0.1/ista_daslab_optimizers.egg-info/requires.txt

@@ -1,9 +0,0 @@
-torch>=2.3.1
-torchaudio>=2.3.1
-torchvision>=0.18.1
-numpy>=1.24.1
-wandb>=0.17.1
-gpustat>=1.1.1
-timm>=1.0.3
-einops>=0.8.0
-psutil>=5.9.8