qqn-torch 0.1.0__tar.gz

qqn_torch-0.1.0/.github/workflows/python-publish.yml

@@ -0,0 +1,80 @@
+# This workflow will upload a Python Package to PyPI when a release is created
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+name: Upload Python Package
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+# Avoid concurrent publish runs for the same release.
+concurrency:
+  group: publish-${{ github.event.release.tag_name }}
+  cancel-in-progress: false
+
+jobs:
+  release-build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.x"
+          cache: "pip"
+
+      - name: Build release distributions
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install build
+          python -m build
+
+      - name: Check distributions with twine
+        run: |
+          python -m pip install twine
+          python -m twine check dist/*
+
+      - name: Upload distributions
+        uses: actions/upload-artifact@v4
+        with:
+          name: release-dists
+          path: dist/
+          if-no-files-found: error
+
+  pypi-publish:
+    runs-on: ubuntu-latest
+    needs:
+      - release-build
+    permissions:
+      # IMPORTANT: this permission is mandatory for trusted publishing
+      id-token: write
+
+    # Dedicated environments with protections for publishing are strongly recommended.
+    # For more information, see: https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#deployment-protection-rules
+    environment:
+      name: pypi
+      url: https://pypi.org/p/qqn-torch
+
+    steps:
+      - name: Retrieve release distributions
+        uses: actions/download-artifact@v4
+        with:
+          name: release-dists
+          path: dist/
+
+      - name: Publish release distributions to PyPI
+        # Trusted publishing (OIDC) is used via the id-token permission above;
+        # no username/password or API token is required.
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: dist/
+          print-hash: true

qqn_torch-0.1.0/.gitignore

@@ -0,0 +1,24 @@
+# Virtual environment
+.venv/
+venv/
+
+# Python junk
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+dist/
+build/
+.idea/
+
+# Testing / coverage
+.pytest_cache/
+.coverage
+htmlcov/
+
+# OS junk
+.DS_Store
+_mnist_data
+profiles/
+#results/
+_fashion_mnist_data

qqn_torch-0.1.0/.llmignore

@@ -0,0 +1,11 @@
+.idea
+.venv
+*.iml
+*.png
+#tests/
+examples/
+experiments/
+*.md
+LICENSE
+results/
+*.js

qqn_torch-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                             Apache License
+                       Version 2.0, January 2004
+                    http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other modifications
+  represent, as a whole, an original work of authorship. For the purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the Work,
+  where such license applies only to those patent claims licensable
+  by such Contributor that are necessarily infringed by their
+  Contribution(s) alone or by combination of their Contribution(s)
+  with the Work to which such Contribution(s) was submitted. If You
+  institute patent litigation against any entity (including a
+  cross-claim or counterclaim in a lawsuit) alleging that the Work
+  or a Contribution incorporated within the Work constitutes direct
+  or contributory patent infringement, then any patent licenses
+  granted to You under this License for that Work shall terminate
+  as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+  Work or Derivative Works thereof in any medium, with or without
+  modifications, and in Source or Object form, provided that You
+  meet the following conditions:
+
+  (a) You must give any other recipients of the Work or
+      Derivative Works a copy of this License; and
+
+  (b) You must cause any modified files to carry prominent notices
+      stating that You changed the files; and
+
+  (c) You must retain, in the Source form of any Derivative Works
+      that You distribute, all copyright, patent, trademark, and
+      attribution notices from the Source form of the Work,
+      excluding those notices that do not pertain to any part of
+      the Derivative Works; and
+
+  (d) If the Work includes a "NOTICE" text file as part of its
+      distribution, then any Derivative Works that You distribute must
+      include a readable copy of the attribution notices contained
+      within such NOTICE file, excluding those notices that do not
+      pertain to any part of the Derivative Works, in at least one
+      of the following places: within a NOTICE text file distributed
+      as part of the Derivative Works; within the Source form or
+      documentation, if provided along with the Derivative Works; or,
+      within a display generated by the Derivative Works, if and
+      wherever such third-party notices normally appear. The contents
+      of the NOTICE file are for informational purposes only and
+      do not modify the License. You may add Your own attribution
+      notices within Derivative Works that You distribute, alongside
+      or as an addendum to the NOTICE text from the Work, provided
+      that such additional attribution notices cannot be construed
+      as modifying the License.
+
+  You may add Your own copyright statement to Your modifications and
+  may provide additional or different license terms and conditions
+  for use, reproduction, or distribution of Your modifications, or
+  for any such Derivative Works as a whole, provided Your use,
+  reproduction, and distribution of the Work otherwise complies with
+  the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+  any Contribution intentionally submitted for inclusion in the Work
+  by You to the Licensor shall be under the terms and conditions of
+  this License, without any additional terms or conditions.
+  Notwithstanding the above, nothing herein shall supersede or modify
+  the terms of any separate license agreement you may have executed
+  with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+  names, trademarks, service marks, or product names of the Licensor,
+  except as required for reasonable and customary use in describing the
+  origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+  agreed to in writing, Licensor provides the Work (and each
+  Contributor provides its Contributions) on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+  implied, including, without limitation, any warranties or conditions
+  of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+  PARTICULAR PURPOSE. You are solely responsible for determining the
+  appropriateness of using or redistributing the Work and assume any
+  risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+  whether in tort (including negligence), contract, or otherwise,
+  unless required by applicable law (such as deliberate and grossly
+  negligent acts) or agreed to in writing, shall any Contributor be
+  liable to You for damages, including any direct, indirect, special,
+  incidental, or consequential damages of any character arising as a
+  result of this License or out of the use or inability to use the
+  Work (including but not limited to damages for loss of goodwill,
+  work stoppage, computer failure or malfunction, or any and all
+  other commercial damages or losses), even if such Contributor
+  has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+  the Work or Derivative Works thereof, You may choose to offer,
+  and charge a fee for, acceptance of support, warranty, indemnity,
+  or other liability obligations and/or rights consistent with this
+  License. However, in accepting such obligations, You may act only
+  on Your own behalf and on Your sole responsibility, not on behalf
+  of any other Contributor, and only if You agree to indemnify,
+  defend, and hold each Contributor harmless for any liability
+  incurred by, or claims asserted against, such Contributor by reason
+  of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+  To apply the Apache License to your work, attach the following
+  boilerplate notice, with the fields enclosed by brackets "[]"
+  replaced with your own identifying information. (Don't include
+  the brackets!)  The text should be enclosed in the appropriate
+  comment syntax for the file format. We also recommend that a
+  file or class name and description of purpose be included on the
+  same "printed page" as the copyright notice for easier
+  identification within third-party archives.
+
+Copyright [yyyy] [name of copyright owner]
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

qqn_torch-0.1.0/PKG-INFO

@@ -0,0 +1,246 @@
+Metadata-Version: 2.4
+Name: qqn-torch
+Version: 0.1.0
+Summary: Quadratic Quasi-Newton optimizer for torch
+Project-URL: Homepage, https://github.com/SimiaCryptus/qqn-torch
+Project-URL: Repository, https://github.com/SimiaCryptus/qqn-torch
+Author: QQN-torch Contributors
+License: Apache License 2.0
+License-File: LICENSE
+Keywords: lbfgs,optimization,qqn,quasi-newton,torch
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Mathematics
+Requires-Python: >=3.9
+Requires-Dist: torch>=1.12
+Provides-Extra: bench
+Requires-Dist: matplotlib>=3.5; extra == 'bench'
+Requires-Dist: torchvision>=0.13; extra == 'bench'
+Provides-Extra: dev
+Requires-Dist: numpy>=1.22; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# qqn-torch
+
+**QQN (Quadratic Quasi-Newton)** — a drop-in replacement for
+`torch.optim.LBFGS` that searches a *quadratic path* blending the
+steepest-descent and quasi-Newton directions.
+
+## What is QQN?
+
+Classic quasi-Newton methods (like L-BFGS) take a single direction `-H∇f`
+and line-search along it. This works well near a minimum but can be fragile
+far from it, where the curvature approximation is unreliable.
+
+QQN instead constructs a **quadratic path** that interpolates between the
+steepest-descent direction and the quasi-Newton direction:
+
+```
+d(t) = t(1-t)·(-∇f) + t²·(-H∇f)
+```
+
+with `t ∈ [0, 1]`. The key properties are:
+
+| `t`           | Behavior                                 |
+|---------------|------------------------------------------|
+| `d(0) = 0`    | The path starts at the current iterate.  |
+| `d'(0) = -∇f` | The initial tangent is steepest descent. |
+| `d(1) = -H∇f` | The endpoint is the L-BFGS direction.    |
+
+Because the path *starts* tangent to `-∇f`, the beginning of the path always
+decreases `f` (when `∇f ≠ 0`). This anchors **global convergence**, while the
+`t = 1` endpoint recovers **L-BFGS superlinear behavior** near the optimum.
+The line search walks `t` directly and *discovers* the right blend — no manual
+tuning of a mixing coefficient.
+
+## Installation
+
+```bash
+pip install qqn-torch
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/your-org/qqn-torch
+cd qqn-torch
+pip install -e .
+```
+
+Requires PyTorch.
+
+## Quick Start
+
+QQN follows the same `closure`-based API as `torch.optim.LBFGS`:
+
+```python
+import torch
+from qqn_torch import QQN
+
+# A simple quadratic objective.
+x = torch.tensor([1.5, -2.0], requires_grad=True)
+
+optimizer = QQN([x], max_iter=20)
+
+
+def closure():
+    optimizer.zero_grad()
+    loss = (x[0] - 3.0) ** 2 + (x[1] + 1.0) ** 2
+    loss.backward()
+    return loss
+
+
+for _ in range(10):
+    loss = optimizer.step(closure)
+    print(f"loss = {loss:.6e}, x = {x.detach().tolist()}")
+```
+
+The `closure` must:
+
+1. Clear gradients (`zero_grad()`),
+2. Compute the loss,
+3. Call `loss.backward()`,
+4. Return the loss.
+
+This is **identical to the `torch.optim.LBFGS` contract**, so existing LBFGS
+training loops work unchanged.
+
+## Configuration
+
+```python
+QQN(
+    params,
+    history_size=10,  # L-BFGS curvature-pair history
+    line_search="armijo",  # "armijo" | "backtracking" | "strong_wolfe" | "fixed"
+    oracle="lbfgs",  # "lbfgs" | "momentum" | "secant" | Oracle instance
+    region=None,  # None | "box" | "trust" | Region instance
+    max_iter=20,  # inner iterations per .step()
+    tol_grad=1e-7,  # gradient-norm stopping tolerance
+    tol_change=1e-9,  # step/objective change tolerance
+    line_search_options=None  # dict forwarded to the line search (c1, c2, ...)
+)
+```
+
+### Four orthogonal, swappable components
+
+QQN is built as a **combiner** of four independent pieces. Each can be
+changed without touching the rest.
+
+#### 1. Oracle — the `t = 1` endpoint (`-H∇f`)
+
+| Name       | Description                                        |
+|------------|----------------------------------------------------|
+| `lbfgs`    | (default) Two-loop recursion over curvature pairs. |
+| `momentum` | Heavy-ball direction `-(β·v + (1-β)·∇f)`.          |
+| `secant`   | Barzilai–Borwein scalar step (`O(n)` memory).      |
+
+Because the steepest-descent contribution anchors convergence, the oracle is
+free to be aggressive — it need not guarantee descent on its own.
+
+#### 2. Line search — walks the path and picks `t`
+
+| Name           | Conditions                 | Notes                            |
+|----------------|----------------------------|----------------------------------|
+| `armijo`       | Armijo sufficient decrease | (default) Backtracking.          |
+| `backtracking` | Armijo sufficient decrease | Aggressive contraction from t=1. |
+| `strong_wolfe` | Armijo + strong curvature  | Can over-restrict the path step. |
+| `fixed`        | None                       | Debug/baseline; constant `t`.    |
+
+The line search is **not** an implementation detail — it is the glue that
+makes the gradient and oracle work together. Convergence quality is bounded
+by line-search quality.
+
+#### 3. Region — optional projection of candidate points
+
+| Name    | Description                               |
+|---------|-------------------------------------------|
+| `None`  | (default) Identity — zero overhead.       |
+| `box`   | Elementwise clip to `[lo, hi]`.           |
+| `trust` | Trust-region sphere with adaptive radius. |
+
+When a region is active, the line search navigates the **projected path**
+`d_R(t) = project_R(x, x + d(t)) - x`, so descent guarantees hold on the
+feasible set. Custom regions can be composed with `SequentialRegion`.
+
+#### 4. Gradient
+
+The raw `-∇f` signal, the path's tangent at the origin.
+
+### Custom components
+
+You can pass instances instead of string shortcuts for full control:
+
+```python
+from qqn_torch import QQN
+from qqn_torch.regions import TrustRegion
+from qqn_torch.oracles import SecantOracle
+
+optimizer = QQN(
+    params,
+    oracle=SecantOracle(alpha_init=0.5),
+    region=TrustRegion(radius=2.0, max_radius=1e3),
+    line_search="strong_wolfe",
+    line_search_options={"c1": 1e-4, "c2": 0.9},
+)
+```
+
+## How it works (per inner iteration)
+
+```
+g       = flat_grad(closure)         # autograd on the closure
+qn_dir  = oracle.direction(g, state) # the t=1 endpoint, -H g
+grad_dir = -g                        # the path tangent at t=0
+d(t)    = t(1-t)·grad_dir + t²·qn_dir # quadratic path
+t*      = line_search(...)           # picks the blend AND the step
+x      += project(d(t*))             # apply (optionally projected) step
+oracle.update(s, y)                  # s = Δx, y = Δg
+region.update(...)                   # e.g. adapt trust radius
+```
+
+## Advantages
+
+- **Adaptive**: automatically balances conservative vs. aggressive steps.
+- **Robust**: `d'(0) = -∇f` plus line-search fallbacks ensure progress even
+  when the oracle is poor.
+- **Efficient**: L-BFGS acceleration when curvature is reliable.
+- **Modular**: gradient, oracle, search, and region are independently
+  swappable.
+
+## Limitations
+
+- **Memory**: stores L-BFGS history (`O(m·n)`).
+- **Overhead**: walking the curved path adds modest per-iteration cost.
+- **Tuning**: sensitive to history size, line-search constants, region radii.
+- **Line-search sensitivity**: a poor line search undermines convergence and
+  the quality of the curvature updates.
+
+## Theoretical guarantees
+
+Under standard assumptions (smooth objective, bounded gradients):
+
+- **Global convergence** — anchored by the steepest-descent tangent.
+- **Superlinear convergence** — inherited from L-BFGS when `t → 1` near the
+  optimum.
+- **Descent property** — every accepted step decreases `f`, enforced by the
+  line search's sufficient-decrease test.
+
+All guarantees are contingent on the line search satisfying sufficient
+decrease. When a region is active, they hold on the projected path `d_R(t)`.
+
+## Documentation
+
+- [`algorithm.md`](algorithm.md) — comprehensive algorithm reference.
+
+## Acknowledgements
+
+The strong-Wolfe line search is adapted from PyTorch's `_strong_wolfe`
+helper (`torch/optim/lbfgs.py`, BSD-licensed). See
+`qqn_torch/_vendor/strong_wolfe.py` for attribution.
+
+## License
+
+See the `LICENSE` file. Vendored code retains its original PyTorch BSD license.