pquant-ml 0.0.1__tar.gz

pquant_ml-0.0.1/.github/workflows/python-publish.yml

@@ -0,0 +1,33 @@
+name: Upload Python Package
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  deploy:
+    environment:
+      name: pypi
+      url: https://pypi.org/p/pquant-ml
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - name: Set up Python
+        uses: actions/setup-python@v3
+        with:
+          python-version: "3.10"
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build
+      - name: Build package
+        run: python -m build
+      - name: Publish package
+        uses: pypa/gh-action-pypi-publish@release/v1

pquant_ml-0.0.1/.gitignore

@@ -0,0 +1,11 @@
+**/__pycache__
+**/data/
+*.txt
+venv/
+**/runs/
+build/
+logs/
+.ipynb_checkpoints/
+*.egg-info
+examples/*.yaml
+**/_version.py

pquant_ml-0.0.1/.pre-commit-config.yaml

@@ -0,0 +1,64 @@
+repos:
+- repo: https://github.com/psf/black
+  rev: 25.1.0
+  hooks:
+  - id: black
+    language_version: python3
+    args: ['--line-length=125',
+           '--skip-string-normalization']
+
+- repo: https://github.com/tox-dev/pyproject-fmt
+  rev: v2.5.1
+  hooks:
+    - id: pyproject-fmt
+
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v5.0.0
+  hooks:
+  - id: check-added-large-files
+  - id: check-case-conflict
+  - id: check-merge-conflict
+  - id: check-symlinks
+  - id: check-toml
+  - id: check-yaml
+  - id: debug-statements
+  - id: end-of-file-fixer
+  - id: mixed-line-ending
+  - id: requirements-txt-fixer
+  - id: trailing-whitespace
+
+- repo: https://github.com/PyCQA/isort
+  rev: 6.0.1
+  hooks:
+  - id: isort
+    args: ["--profile=black"] # <-- this one
+
+- repo: https://github.com/asottile/pyupgrade
+  rev: v3.19.1
+  hooks:
+  - id: pyupgrade
+    args: ["--py36-plus"]
+
+- repo: https://github.com/pycqa/flake8
+  rev: 7.1.2
+  hooks:
+  - id: flake8
+    exclude: docs/conf.py
+    additional_dependencies: [flake8-bugbear, flake8-print]
+    args: ['--max-line-length=125',  # github viewer width
+           '--extend-ignore=E203']  # E203 is not PEP8 compliant
+
+- repo: https://github.com/mgedmin/check-manifest
+  rev: "0.50"
+  hooks:
+  - id: check-manifest
+    stages: [manual]
+
+ci:
+    autofix_commit_msg: '[pre-commit.ci] auto fixes from pre-commit hooks'
+    autofix_prs: false  # default is true
+    autoupdate_branch: 'main'
+    autoupdate_commit_msg: '[pre-commit.ci] pre-commit autoupdate'
+    autoupdate_schedule: weekly
+    skip: []
+    submodules: true

pquant_ml-0.0.1/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright {yyyy} {name of copyright owner}
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

pquant_ml-0.0.1/PKG-INFO

@@ -0,0 +1,68 @@
+Metadata-Version: 2.4
+Name: pquant-ml
+Version: 0.0.1
+Summary: Pruning and Quantization of ML models
+Author: NGT 1.3 team
+License: Apache-2.0
+Project-URL: repository, https://github.com/nroope/PQuant
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: keras>=3
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: quantizers>=1.1
+Requires-Dist: torch>=2.1
+Dynamic: license-file
+
+![alt text](docs/_static/pquant.png)
+
+## Prune and Quantize ML models
+PQuant is a library for training compressed machine learning models, developed at CERN as part of the [Next Generation Triggers](https://nextgentriggers.web.cern.ch/t13/) project.
+
+PQuant replaces the layers and activations it finds with a Compressed (in the case of layers) or Quantized (in the case of activations) variant. These automatically handle the quantization of the weights, biases and activations, and the pruning of the weights. 
+Both PyTorch and TensorFlow models are supported. 
+
+Layers that can be compressed: Conv2D and Linear layers, Tanh and ReLU activations for both TensorFlow and PyTorch. For PyTorch, also Conv1D.
+
+![alt text](docs/_static/pquant_transform.png)
+
+The various pruning methods have different training steps, such as a pre-training step and fine-tuning step. PQuant provides a training function, where the user provides the functions to train and validate an epoch, and PQuant handles the training while triggering the different training steps.
+
+
+
+### Example
+Example notebook can be found [here](https://github.com/nroope/PQuant/tree/main/examples). It handles the
+  1. Creation of a torch model and data loaders.
+  2. Creation of the training and validation functions.
+  3. Loading a default pruning configuration of a pruning method.
+  4. Using the configuration, the model, and the training and validation functions, call the training function of PQuant to train and compress the model.
+  5. Creating a custom quantization and pruning configuration for a given model (disable pruning for some layers, different quantization bitwidths for different layers).
+
+### Pruning methods
+A description of the pruning methods and their hyperparameters can be found [here](docs/pruning_methods.md).
+
+### Quantization parameters
+A description of the quantization parameters can be found [here](docs/quantization_parameters.md).
+
+
+### Installation
+
+```pip install .``` for regular install, ```pip install -e .``` if you wish to install as a local editable package
+To run the code, [HGQ2](https://github.com/calad0i/HGQ2) is also needed. For now it only has local install available, so download the repository and install it locally.
+
+### Authors
+ - Roope Niemi (CERN)
+ - Anastasiia Petrovych (CERN)
+ - Chang Sun (Caltech)
+ - Michael Kagan (SLAC National Accelerator Laboratory)
+ - Vladimir Loncar (CERN)

pquant_ml-0.0.1/README.md

@@ -0,0 +1,42 @@
+![alt text](docs/_static/pquant.png)
+
+## Prune and Quantize ML models
+PQuant is a library for training compressed machine learning models, developed at CERN as part of the [Next Generation Triggers](https://nextgentriggers.web.cern.ch/t13/) project.
+
+PQuant replaces the layers and activations it finds with a Compressed (in the case of layers) or Quantized (in the case of activations) variant. These automatically handle the quantization of the weights, biases and activations, and the pruning of the weights. 
+Both PyTorch and TensorFlow models are supported. 
+
+Layers that can be compressed: Conv2D and Linear layers, Tanh and ReLU activations for both TensorFlow and PyTorch. For PyTorch, also Conv1D.
+
+![alt text](docs/_static/pquant_transform.png)
+
+The various pruning methods have different training steps, such as a pre-training step and fine-tuning step. PQuant provides a training function, where the user provides the functions to train and validate an epoch, and PQuant handles the training while triggering the different training steps.
+
+
+
+### Example
+Example notebook can be found [here](https://github.com/nroope/PQuant/tree/main/examples). It handles the
+  1. Creation of a torch model and data loaders.
+  2. Creation of the training and validation functions.
+  3. Loading a default pruning configuration of a pruning method.
+  4. Using the configuration, the model, and the training and validation functions, call the training function of PQuant to train and compress the model.
+  5. Creating a custom quantization and pruning configuration for a given model (disable pruning for some layers, different quantization bitwidths for different layers).
+
+### Pruning methods
+A description of the pruning methods and their hyperparameters can be found [here](docs/pruning_methods.md).
+
+### Quantization parameters
+A description of the quantization parameters can be found [here](docs/quantization_parameters.md).
+
+
+### Installation
+
+```pip install .``` for regular install, ```pip install -e .``` if you wish to install as a local editable package
+To run the code, [HGQ2](https://github.com/calad0i/HGQ2) is also needed. For now it only has local install available, so download the repository and install it locally.
+
+### Authors
+ - Roope Niemi (CERN)
+ - Anastasiia Petrovych (CERN)
+ - Chang Sun (Caltech)
+ - Michael Kagan (SLAC National Accelerator Laboratory)
+ - Vladimir Loncar (CERN)

pquant_ml-0.0.1/docs/pruning_methods.md

@@ -0,0 +1,111 @@
+# Descriptions of the pruning methods
+
+Our implementations follow the actual implementations of the author's of the papers, whenever we were able to find one. Because of this some of the functionality of the pruning methods can differ slightly from the equations shown in the papers.
+
+#### [Activation pruning](https://arxiv.org/abs/1903.04476)
+Collect layer outputs to calculate average layer activity (how often layer neuron / channel outputs values greater than 0). Prune those neurons and channels which have smaller activity value than a given threshold.
+
+**Hyperparameters**
+- `threshold`: If a neuron or channel is less active than this threshold, prune it.
+- `threshold_decay`: Not used.
+- `t_delta`: How many batches to collect as calibration data.
+- `t_start_collecting_batch`: At which epoch during training the collection begins
+
+#### [AutoSparse](https://arxiv.org/abs/2304.06941)
+$x = sign(W) \cdot ReLU(|W| - \sigma(T))$.
+```math
+g = \begin{cases}
+    1, & \text{if W > 0} \\
+    \alpha, & \text{otherwise}\quad,
+\end{cases}
+```
+where T is threshold, W is the weight matrix, g is the gradient.
+ $\alpha$ is decayed after each epoch using cosine sigmoid decay.
+
+**Hyperparameters:**
+- `alpha`: initial value for $\alpha$
+- `backward_sparsity`: if true, sets gradients to 0 for weights in the bottom 50% magnitude of weights in the layer. False in the default config.
+- `threshold_decay`: threshold decay for optimizer. 0 in the default config.
+- `threshold_init`: initial value for threshold. -5 in the default config.
+- `threshold_type`: weightwise/channelwise/layerwise. Defines whether each weight has its own threshold, or is threshold shared between weights in a channel, or does the whole layer have one threshold.
+
+#### [Continuous Sparsification](https://arxiv.org/abs/1912.04427)
+A multi-round pruning algorithm.
+```math
+ x = W\cdot M
+```
+where
+```math
+M=(\frac{\sigma(\beta s)}{\sigma(s_{init})})
+```
+$\beta$ starts from the initial value at the beginning of each round, and increased exponentially until reaching a final value. $s$ is a learnable matrix with a same shape as the weight matrix. $s_{init}$ is the initial value of $s$.
+
+During each round, as the $s$ matrix is learning and the $\beta$ is increased, the values of the mask get pushed more and more towards 0 and 1. After each round, $\beta$ is reset, and the positive values of $s$ are set to $s_{init}$ value, and negative values are kept as they are. This means that the weights pruned by $s$ stay pruned after each round, but the weights that have not been pruned previously can be pruned after a new round begins, since their values are reset in $s$.
+
+Before fine-tuning the mask is fixed and converted to a hard mask of 0s and 1s, and all the weights rewinded back to an earlier state.
+
+**Hyperparameters**
+- `final_temp`: Value up to which $\beta$ is increased during each round. 200 in the default config.
+- `threshold_decay`: L1 decay for the $s$ matrix. 1.0e-09 in the default config.
+- `threshold_init`: Initial value for $s$. 0 in the default config. Lower value means more pruning, higher value means less pruning.
+
+
+#### [DST](https://arxiv.org/abs/2005.06870)
+$x = ReLU(|W| - T)$.
+```math
+g = \begin{cases}
+    2-4\cdot|W|, & \text{if } |x| \leq 0.4 \\
+    0.4, & \text{if } 0.4 < |x| \leq 1 \\
+    0, & \text{if }|x| > 1\quad.
+\end{cases}
+```
+The threshold T is controlled by additional loss, which is calculated by
+```math
+\alpha \cdot \sum_{i,j}{e^{-T_{i,j}}}
+```
+
+**Hyperparameters**
+- `alpha`: Used to control the threshold via loss. 5.0e-06 in the default config.
+- `max_pruning_pct`: The algorithm has a tendency to prune whole layers, so if pruning goes higher than this value, reset the threshold. 0.99 in the default config.
+- `threshold_decay`: threshold decay for optimizer. 0 in the default config.
+- `threshold_init`: Initial value for threshold. 0 in the default config.
+- `threshold_type`: weightwise/channelwise/layerwise. Defines whether each weight has its own threshold, or is threshold shared between weights in a channel, or does the whole layer have one threshold.
+
+
+
+#### [PDP](https://arxiv.org/abs/2305.11203)
+Captures weight distribution of each layer and calculates a threshold, then does a softmax between the weights and this value, creating a soft mask.
+
+
+$`W_h = topK(|W|, (1-r) \cdot n(W))\newline`$\
+$`W_i = bottomK(|W|, r \cdot n(W))`$\
+$`t = 0.5 \cdot (min(W_h) + max(W_i))`$\
+$`zw, mw = softmax(\frac{t^2, w^2}{\tau})\text{ for $w$ in $W$}`$\
+$`w = mw \cdot w`$,
+
+where $\tau$ is the temperature, $r$ is the target sparsity of the layer for that iteration, $n(W)$ is the number of weights. The $mw$ in the above equation will have all the softmax values of the weights between the weight tensor and the threshold. If a weight is above the threshold, due to the temperature, the softmax result will very quickly go towards 1. The $r$ is increased linearly during training. The layerwise budget sparsity is calculated after a pre-training phase, in a way that the total sparsity of the model is the target sparsity given in the config.
+
+PDP has an unstructured, N:M pruning (not yet implemented here), and channel pruning version.
+
+**Hyperparameters**
+  `epsilon`: How fast to increaes the sparsity during training. After each epoch, the sparsity is increased by this amount, until the value reaches 1 (100% of target sparsity). 0.015 in the default config, which means after ~70 epochs the target sparsity has been reached.
+- `sparsity`: Target sparsity for the whole model
+- `temperature`: Temperature of the softmax. 1e-5 in the default config
+- `threshold_decay`: Not used
+- `structured_pruning`: Whether to use a structured pruning variant or not. Structured pruning uses l2 norms of the channels/neurons instead of absolute values of weights when calculating the threshold, and prunes whole channels/neurons using that threshold value.
+
+#### [Wanda](https://arxiv.org/abs/2306.11695)
+One shot pruning, originally a post-training pruning method without fine-tuning (to implement the post-training version is on the to-do list).
+
+Using a calibration data set, calculate a metric based on the average input to the layer, and multiply the absolute values of the weights with that metric. Prune weights based on this multiplication result (lowest values being pruned first), until a target sparsity has been reached.
+
+For linear layers, the metric is calculated as L2 norm over the batch dimension. For convolutions, reduce dimensions by taking the average of the batch dimension, then calculate L2 norm over a flattened kernel dimension.
+
+**Hyperparameters**
+- `calculate_pruning_budget`: If True, calculate the pruning budget for each layer, while keeping the target sparsity. If False, prunes every layer using target sparsity.
+- `M`: If doing N:M pruning, N and M should be non-null (N < M)
+- `N`: If doing N:M pruning, N and M should be non-null (N < M)
+- `threshold_decay`: not used
+- `sparsity`: target sparsity. 0.9 in the default config
+- `t_delta`: how many batches to collect as calibration data
+- `t_start_collecting`: training step when collection starts

pquant_ml-0.0.1/docs/quantization_parameters.md

@@ -0,0 +1,11 @@
+# Descriptions of the quantization parameters
+ - `default_integer_bits`: Default integer bits used for quantization
+ - `default_fractional_bits`: Default fractional bits used for quantization. For ReLU, because it is unsigned and no bits are used for the sign, 1 bit is added to the default value during adding of compression layers.
+ - `enable_quantization`: Enables quantization
+ - `hgq_gamma`: scales the loss of HGQ. If too high, can prune the whole model.
+ - `hgq_heterogeneous`: If true, HGQ learns one set of bits for each weight in the model. If false, learns one set of bits for each layer in the model
+ - `layer_specific`: Layers that use non-default quantization bits, should be added here. A default config with all the layers can be created using the function `pquant.add_default_layer_quantization_pruning_to_config`
+ - `use_high_granularity_quantization`: If true, uses HGQ instead of fixed quantizers
+ - `use_real_tanh`: If true, use real tanh function before quantization. If false, uses hard tanh
+ - `use_relu_multiplier`: If true, multiply the input of QuantizedReLU with a learned multiplier before the QuantizedReLU operation. The multiplication operation will be `inputs_to_relu = inputs_to_relu * 2 ** (round(learned_multiplier))`. Learned multiplier is initialized at -1, therefore at the beginning of the training the inputs are multiplied by 0.5 before QuantizedReLU.
+ - `use_symmetric_quantization`: if true, `minimum_quantized_value == -maximum_quantized_value`