gbra 0.1.1__tar.gz

gbra-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Berke T. Adanc (GEMBER && Apex Logic Studio)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gbra-0.1.1/PKG-INFO

@@ -0,0 +1,108 @@
+Metadata-Version: 2.4
+Name: gbra
+Version: 0.1.1
+Summary: GBRA (Gember-Berke Rank Adaptation): DoRA-style fine-tuning with a buffered, lazily refreshed weight norm.
+Author: Berke T. Adanc
+License: MIT
+Keywords: lora,dora,peft,fine-tuning,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.0
+Requires-Dist: safetensors>=0.4
+Provides-Extra: hf
+Requires-Dist: transformers>=4.40; extra == "hf"
+Provides-Extra: 4bit
+Requires-Dist: bitsandbytes>=0.43; extra == "4bit"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: transformers>=4.40; extra == "test"
+Dynamic: license-file
+
+# GBRA: Gember-Berke Rank Adaptation
+
+*Pronounced "Gember RA". A GEMBER && Apex Logiç Studio project by Berke T. Adanç.*
+
+GBRA is a DoRA-style parameter-efficient fine-tuning method. It keeps DoRA's split of each weight into a learned **magnitude** and a low-rank-updated **direction**, but **buffers the weight norm** instead of recomputing it in every forward pass.
+
+```
+DoRA:  W' = m · (W + s·BA) / ‖W + s·BA‖     ← norm recomputed every forward (and again during gradient-checkpoint recompute)
+GBRA:  W' = m · (W + s·BA) / n               ← n is a stored buffer, refreshed every K optimizer steps
+```
+
+**Why it helps:** With gradient accumulation, the weights only change once per optimizer step. Recomputing the norm for every micro-batch repeats the same work many times. GBRA computes it once and reuses it.
+
+## Install
+
+```bash
+pip install gbra             # core
+pip install "gbra[hf]"       # + Hugging Face Trainer callback
+pip install "gbra[4bit]"     # + 4-bit (bitsandbytes) base models
+```
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, Trainer
+from gbra import apply_gbra, GBRACallback, save_gbra, merge_gbra
+
+model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
+model = apply_gbra(model, r=16, alpha=16)          # wraps q/k/v/o/gate/up/down projections
+
+trainer = Trainer(model=model, ..., callbacks=[GBRACallback(refresh_every=10)])
+trainer.train()
+
+save_gbra(model, "my_adapter")                     # small adapter (A, B, m, n)
+merge_gbra(model)                                  # fold into plain nn.Linear for export
+```
+
+Turkish aliases are also available: `gbra_ekle`, `tum_normlari_yenile`, `gbra_kaydet`, `gbra_yukle`, `gbra_birlestir`.
+
+## Preliminary results
+
+Qwen2.5-0.5B-Instruct, 3,000 Turkish chat/tool-use examples, 375 optimizer steps (batch 1 × 8 accumulation), r=16, single Kaggle T4, fp16 autocast. Same data, seed and hyper-parameters for every method. DoRA and LoRA via 🤗 PEFT.
+
+| Method | s / step | vs LoRA | Peak memory | Val. loss |
+|---|---|---|---|---|
+| LoRA | **5.77** | 1.00× | 8.11 GB | 1.6376 |
+| DoRA (PEFT) | 9.10 | 1.58× | 8.11 GB | 1.6285 |
+| **GBRA K=10** | 6.47 | 1.12× | 8.11 GB | **1.6249** |
+| GBRA K=100 | 6.47 | 1.12× | 8.11 GB | 1.6253 |
+| GBRA K=never | 6.46 | 1.12× | 8.11 GB | 1.6263 |
+
+**In this run, GBRA was ~29% faster than DoRA per step, with equal or slightly lower validation loss.** A shorter 125-step run gave the same speed ratio.
+
+### Limitations (please read)
+
+- **Single seed, single small model, single dataset.** The loss differences between GBRA variants and DoRA are small (≈0.001–0.004) and may be noise; repeated runs with more seeds are in progress.
+- Much of the speed-up comes from **not recomputing the norm per micro-batch**. With no gradient accumulation and no gradient checkpointing, the gap to DoRA should be smaller.
+- The comparison is against PEFT's DoRA implementation at the time of testing, not against every possible DoRA implementation.
+- 4-bit (bitsandbytes) support is implemented but **has not yet been benchmarked**.
+- Unlike PEFT LoRA, GBRA does not currently use fused kernels such as Unsloth's.
+
+## Credits
+
+GBRA builds directly on **DoRA** (Liu et al., 2024, *"DoRA: Weight-Decomposed Low-Rank Adaptation"*) and **LoRA** (Hu et al., 2021). Related ideas on learned scaling vectors appear in (IA)³ and VeRA. The contribution here is the buffered / lazily refreshed norm and its evaluation.
+
+The core `forward` of `GBRALinear` was written by Berke T. Adanç. Parts of the surrounding library and tests were developed with the help of an AI assistant.
+
+## Changelog
+
+- **0.1.1**: Python 3.9 compatibility fix in `GBRACallback`; new `refresh_on_end` option (refreshes the norm once more when training ends); README install instructions for PyPI.
+- **0.1.0**: First release.
+
+## License
+
+MIT
+
+---
+
+## Türkçe özet
+
+GBRA, DoRA'nın "büyüklük + yön" ayrımını korur, ama ağırlık normunu her adımda yeniden hesaplamak yerine **saklar** ve sadece **K optimizer adımında bir** günceller. Gradient birikimi kullanılan eğitimde ağırlıklar mini-batch'ler arasında değişmediği için DoRA aynı normu tekrar tekrar hesaplar. GBRA bu tekrarı ortadan kaldırır.
+
+İlk deneyde (Qwen2.5-0.5B, 375 adım, tek seed) GBRA, DoRA'dan **~%29 daha hızlı** ve doğrulama loss'u aynı ya da biraz daha iyiydi. Sonuçlar tek seed'e dayandığı için **ön sonuç** olarak değerlendirilmelidir.

gbra-0.1.1/README.md

@@ -0,0 +1,83 @@
+# GBRA: Gember-Berke Rank Adaptation
+
+*Pronounced "Gember RA". A GEMBER && Apex Logiç Studio project by Berke T. Adanç.*
+
+GBRA is a DoRA-style parameter-efficient fine-tuning method. It keeps DoRA's split of each weight into a learned **magnitude** and a low-rank-updated **direction**, but **buffers the weight norm** instead of recomputing it in every forward pass.
+
+```
+DoRA:  W' = m · (W + s·BA) / ‖W + s·BA‖     ← norm recomputed every forward (and again during gradient-checkpoint recompute)
+GBRA:  W' = m · (W + s·BA) / n               ← n is a stored buffer, refreshed every K optimizer steps
+```
+
+**Why it helps:** With gradient accumulation, the weights only change once per optimizer step. Recomputing the norm for every micro-batch repeats the same work many times. GBRA computes it once and reuses it.
+
+## Install
+
+```bash
+pip install gbra             # core
+pip install "gbra[hf]"       # + Hugging Face Trainer callback
+pip install "gbra[4bit]"     # + 4-bit (bitsandbytes) base models
+```
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, Trainer
+from gbra import apply_gbra, GBRACallback, save_gbra, merge_gbra
+
+model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
+model = apply_gbra(model, r=16, alpha=16)          # wraps q/k/v/o/gate/up/down projections
+
+trainer = Trainer(model=model, ..., callbacks=[GBRACallback(refresh_every=10)])
+trainer.train()
+
+save_gbra(model, "my_adapter")                     # small adapter (A, B, m, n)
+merge_gbra(model)                                  # fold into plain nn.Linear for export
+```
+
+Turkish aliases are also available: `gbra_ekle`, `tum_normlari_yenile`, `gbra_kaydet`, `gbra_yukle`, `gbra_birlestir`.
+
+## Preliminary results
+
+Qwen2.5-0.5B-Instruct, 3,000 Turkish chat/tool-use examples, 375 optimizer steps (batch 1 × 8 accumulation), r=16, single Kaggle T4, fp16 autocast. Same data, seed and hyper-parameters for every method. DoRA and LoRA via 🤗 PEFT.
+
+| Method | s / step | vs LoRA | Peak memory | Val. loss |
+|---|---|---|---|---|
+| LoRA | **5.77** | 1.00× | 8.11 GB | 1.6376 |
+| DoRA (PEFT) | 9.10 | 1.58× | 8.11 GB | 1.6285 |
+| **GBRA K=10** | 6.47 | 1.12× | 8.11 GB | **1.6249** |
+| GBRA K=100 | 6.47 | 1.12× | 8.11 GB | 1.6253 |
+| GBRA K=never | 6.46 | 1.12× | 8.11 GB | 1.6263 |
+
+**In this run, GBRA was ~29% faster than DoRA per step, with equal or slightly lower validation loss.** A shorter 125-step run gave the same speed ratio.
+
+### Limitations (please read)
+
+- **Single seed, single small model, single dataset.** The loss differences between GBRA variants and DoRA are small (≈0.001–0.004) and may be noise; repeated runs with more seeds are in progress.
+- Much of the speed-up comes from **not recomputing the norm per micro-batch**. With no gradient accumulation and no gradient checkpointing, the gap to DoRA should be smaller.
+- The comparison is against PEFT's DoRA implementation at the time of testing, not against every possible DoRA implementation.
+- 4-bit (bitsandbytes) support is implemented but **has not yet been benchmarked**.
+- Unlike PEFT LoRA, GBRA does not currently use fused kernels such as Unsloth's.
+
+## Credits
+
+GBRA builds directly on **DoRA** (Liu et al., 2024, *"DoRA: Weight-Decomposed Low-Rank Adaptation"*) and **LoRA** (Hu et al., 2021). Related ideas on learned scaling vectors appear in (IA)³ and VeRA. The contribution here is the buffered / lazily refreshed norm and its evaluation.
+
+The core `forward` of `GBRALinear` was written by Berke T. Adanç. Parts of the surrounding library and tests were developed with the help of an AI assistant.
+
+## Changelog
+
+- **0.1.1**: Python 3.9 compatibility fix in `GBRACallback`; new `refresh_on_end` option (refreshes the norm once more when training ends); README install instructions for PyPI.
+- **0.1.0**: First release.
+
+## License
+
+MIT
+
+---
+
+## Türkçe özet
+
+GBRA, DoRA'nın "büyüklük + yön" ayrımını korur, ama ağırlık normunu her adımda yeniden hesaplamak yerine **saklar** ve sadece **K optimizer adımında bir** günceller. Gradient birikimi kullanılan eğitimde ağırlıklar mini-batch'ler arasında değişmediği için DoRA aynı normu tekrar tekrar hesaplar. GBRA bu tekrarı ortadan kaldırır.
+
+İlk deneyde (Qwen2.5-0.5B, 375 adım, tek seed) GBRA, DoRA'dan **~%29 daha hızlı** ve doğrulama loss'u aynı ya da biraz daha iyiydi. Sonuçlar tek seed'e dayandığı için **ön sonuç** olarak değerlendirilmelidir.

gbra-0.1.1/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "gbra"
+version = "0.1.1"
+description = "GBRA (Gember-Berke Rank Adaptation): DoRA-style fine-tuning with a buffered, lazily refreshed weight norm."
+readme = "README.md"
+license = { text = "MIT" }
+authors = [{ name = "Berke T. Adanc" }]
+requires-python = ">=3.9"
+dependencies = ["torch>=2.0", "safetensors>=0.4"]
+keywords = ["lora", "dora", "peft", "fine-tuning", "llm"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Programming Language :: Python :: 3",
+  "License :: OSI Approved :: MIT License",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+
+[project.optional-dependencies]
+hf = ["transformers>=4.40"]
+4bit = ["bitsandbytes>=0.43"]
+test = ["pytest", "transformers>=4.40"]
+
+[tool.setuptools.packages.find]
+where = ["src"]

gbra-0.1.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

gbra-0.1.1/src/gbra/__init__.py

@@ -0,0 +1,29 @@
+"""GBRA - Gember-Berke Rank Adaptation ("Gember RA").
+
+DoRA-style fine-tuning with a buffered (lazily refreshed) weight norm.
+GEMBER && Apex Logic Studio - Berke T. Adanc
+"""
+
+from .layer import GBRALinear
+from .model import (
+    DEFAULT_TARGETS,
+    apply_gbra, refresh_norms, save_gbra, load_gbra, merge_gbra, gbra_state_dict,
+    gbra_ekle, tum_normlari_yenile, gbra_kaydet, gbra_yukle, gbra_birlestir,
+)
+
+__version__ = "0.1.1"
+
+
+def __getattr__(name):
+    # GBRACallback needs transformers; import it lazily so the core works without it.
+    if name == "GBRACallback":
+        from .callback import GBRACallback
+        return GBRACallback
+    raise AttributeError(name)
+
+
+__all__ = [
+    "GBRALinear", "GBRACallback", "DEFAULT_TARGETS",
+    "apply_gbra", "refresh_norms", "save_gbra", "load_gbra", "merge_gbra", "gbra_state_dict",
+    "gbra_ekle", "tum_normlari_yenile", "gbra_kaydet", "gbra_yukle", "gbra_birlestir",
+]

gbra-0.1.1/src/gbra/callback.py

@@ -0,0 +1,32 @@
+"""Hugging Face Trainer integration."""
+
+from typing import Optional
+
+from transformers import TrainerCallback
+
+from .model import refresh_norms
+
+
+class GBRACallback(TrainerCallback):
+    """Refreshes GBRA norms every `refresh_every` optimizer steps (K).
+
+    Use refresh_every=None to never refresh during training (the norm stays at its
+    initial value). With refresh_on_end=True (default) the norm is refreshed once more
+    when training finishes, so a saved adapter always carries an up-to-date norm.
+
+        trainer = Trainer(..., callbacks=[GBRACallback(refresh_every=10)])
+    """
+
+    def __init__(self, refresh_every: Optional[int] = 10, refresh_on_end: bool = True):
+        if refresh_every is not None and refresh_every < 1:
+            raise ValueError("refresh_every must be >= 1 or None")
+        self.k = refresh_every
+        self.refresh_on_end = refresh_on_end
+
+    def on_step_end(self, args, state, control, model=None, **kwargs):
+        if self.k is not None and model is not None and state.global_step % self.k == 0:
+            refresh_norms(model)
+
+    def on_train_end(self, args, state, control, model=None, **kwargs):
+        if self.refresh_on_end and model is not None:
+            refresh_norms(model)

gbra-0.1.1/src/gbra/layer.py

@@ -0,0 +1,97 @@
+"""GBRALinear: the core GBRA layer.
+
+GBRA (Gember-Berke Rank Adaptation, read "Gember RA") keeps DoRA's
+magnitude/direction split but buffers the column-norm instead of
+recomputing it on every forward pass:
+
+    DoRA:  W' = m * (W + s*B@A) / ||W + s*B@A||     (norm recomputed every forward)
+    GBRA:  W' = m * (W + s*B@A) / n                 (n is a buffer, refreshed every K optimizer steps)
+
+Norms are taken per output row (W has shape [out, in]).
+"""
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+def _is_4bit(layer: nn.Module) -> bool:
+    return hasattr(getattr(layer, "weight", None), "quant_state")
+
+
+def base_weight(layer: nn.Linear) -> torch.Tensor:
+    """Return the base weight as a dense float32 tensor (dequantizes 4-bit layers)."""
+    if _is_4bit(layer):
+        import bitsandbytes.functional as bnbF  # optional dependency
+        return bnbF.dequantize_4bit(layer.weight.data, layer.weight.quant_state).float()
+    return layer.weight.detach().float()
+
+
+class GBRALinear(nn.Module):
+    def __init__(self, taban: nn.Linear, r: int = 16, alpha: int = 16):
+        super().__init__()
+        if r <= 0:
+            raise ValueError("r must be a positive integer")
+        self.taban = taban                      # original layer (frozen)
+        for p in self.taban.parameters():
+            p.requires_grad = False
+
+        cikis, giris = taban.out_features, taban.in_features
+        self.r, self.alpha = r, alpha
+        self.s = alpha / r                      # LoRA scale
+        device = taban.weight.device
+
+        # LoRA matrices. B starts at zero, so at init GBRA == the original layer.
+        self.A = nn.Parameter(torch.empty(r, giris, device=device, dtype=torch.float32))
+        self.B = nn.Parameter(torch.zeros(cikis, r, device=device, dtype=torch.float32))
+        nn.init.kaiming_uniform_(self.A, a=5 ** 0.5)
+
+        # Magnitude vector m (learned), initialised to the row norm of W.
+        with torch.no_grad():
+            norm0 = base_weight(taban).norm(dim=1)
+        self.m = nn.Parameter(norm0.clone())
+
+        # Buffered norm n (NOT learned). Updated only by normu_yenile().
+        self.register_buffer("n", norm0.clone())
+
+    @torch.no_grad()
+    def normu_yenile(self):
+        """Lazy norm refresh: compute ||W + s*B@A|| for real and store it in n."""
+        W = base_weight(self.taban)
+        delta = self.s * (self.B.float() @ self.A.float())
+        self.n.copy_((W + delta).norm(dim=1))
+
+    refresh_norm = normu_yenile  # English alias
+
+    def _ana(self, x):
+        """x @ W.T without bias (works for normal and 4-bit layers)."""
+        if _is_4bit(self.taban):
+            y = self.taban(x)
+            if self.taban.bias is not None:
+                y = y - self.taban.bias
+            return y
+        return F.linear(x, self.taban.weight, bias=None)
+
+    def forward(self, x):
+        # forward: Berke T. Adanc
+        # 1. Ana ağırlık: x @ W.T (bias eklemeden)
+        ana = self._ana(x)
+
+        # 2. LoRA: önce A, sonra B
+        lora_a = F.linear(x, self.A.to(x.dtype), bias=None)
+        lora_b = F.linear(lora_a, self.B.to(x.dtype), bias=None)
+
+        # 3. Ölçek: m / n, ardından x'in dtype'ına dönüştür
+        olcek = (self.m / self.n).to(x.dtype)
+
+        # 4. Ana ağırlık + LoRA toplamını ölçekle
+        y = olcek * (ana + self.s * lora_b)
+
+        # 5. Bias'ı en son ekle (None olabilir)
+        if self.taban.bias is not None:
+            y = y + self.taban.bias
+
+        return y
+
+    def extra_repr(self):
+        return f"in={self.taban.in_features}, out={self.taban.out_features}, r={self.r}, alpha={self.alpha}"

gbra-0.1.1/src/gbra/model.py

@@ -0,0 +1,123 @@
+"""Model-level helpers: attach GBRA, refresh norms, merge, save and load."""
+
+import json
+import os
+
+import torch
+import torch.nn as nn
+
+from .layer import GBRALinear, base_weight
+
+DEFAULT_TARGETS = ("q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj")
+CONFIG_NAME = "gbra_config.json"
+WEIGHTS_NAME = "gbra_adapter.safetensors"
+
+
+def apply_gbra(model: nn.Module, targets=DEFAULT_TARGETS, r: int = 16, alpha: int = 16,
+               verbose: bool = True) -> nn.Module:
+    """Freeze the model and wrap every target nn.Linear (incl. 4-bit) with GBRALinear."""
+    for p in model.parameters():
+        p.requires_grad = False
+    count = 0
+    for module in list(model.modules()):
+        for name, child in list(module.named_children()):
+            if name in targets and isinstance(child, nn.Linear) and not isinstance(child, GBRALinear):
+                setattr(module, name, GBRALinear(child, r=r, alpha=alpha))
+                count += 1
+    if count == 0:
+        raise ValueError(f"No target layers found. Check the layer names of your model. targets={targets}")
+    model._gbra_config = {"r": r, "alpha": alpha, "targets": list(targets)}
+    if verbose:
+        trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
+        total = sum(p.numel() for p in model.parameters())
+        print(f"GBRA: {count} layers | trainable {trainable:,} / {total:,} ({100 * trainable / total:.2f}%)")
+    return model
+
+
+@torch.no_grad()
+def refresh_norms(model: nn.Module) -> int:
+    """Refresh the buffered norm of every GBRA layer. Returns the number of layers."""
+    n = 0
+    for m in model.modules():
+        if isinstance(m, GBRALinear):
+            m.normu_yenile()
+            n += 1
+    return n
+
+
+def gbra_state_dict(model: nn.Module) -> dict:
+    """Only the GBRA tensors (A, B, m, n) - small enough to share as an adapter."""
+    out = {}
+    for name, m in model.named_modules():
+        if isinstance(m, GBRALinear):
+            for key in ("A", "B", "m", "n"):
+                out[f"{name}.{key}"] = getattr(m, key).detach().cpu().contiguous()
+    return out
+
+
+def save_gbra(model: nn.Module, path: str):
+    os.makedirs(path, exist_ok=True)
+    cfg = getattr(model, "_gbra_config", None)
+    if cfg is None:
+        raise ValueError("This model has no GBRA layers (call apply_gbra first).")
+    with open(os.path.join(path, CONFIG_NAME), "w", encoding="utf-8") as f:
+        json.dump({"format": "gbra", "version": 1, **cfg}, f, indent=2)
+    from safetensors.torch import save_file
+    save_file(gbra_state_dict(model), os.path.join(path, WEIGHTS_NAME))
+
+
+def load_gbra(model: nn.Module, path: str) -> nn.Module:
+    """Attach GBRA to a fresh base model and load adapter weights saved with save_gbra."""
+    with open(os.path.join(path, CONFIG_NAME), encoding="utf-8") as f:
+        cfg = json.load(f)
+    if not any(isinstance(m, GBRALinear) for m in model.modules()):
+        apply_gbra(model, targets=tuple(cfg["targets"]), r=cfg["r"], alpha=cfg["alpha"], verbose=False)
+    from safetensors.torch import load_file
+    tensors = load_file(os.path.join(path, WEIGHTS_NAME))
+    modules = dict(model.named_modules())
+    used = set()
+    for full, t in tensors.items():
+        name, key = full.rsplit(".", 1)
+        layer = modules.get(name)
+        if not isinstance(layer, GBRALinear):
+            raise KeyError(f"Adapter tensor {full} has no matching GBRA layer in this model.")
+        getattr(layer, key).data.copy_(t.to(getattr(layer, key).device))
+        used.add(name)
+    missing = [n for n, m in modules.items() if isinstance(m, GBRALinear) and n not in used]
+    if missing:
+        raise KeyError(f"No adapter weights for {len(missing)} GBRA layers, e.g. {missing[0]}")
+    return model
+
+
+@torch.no_grad()
+def merge_gbra(model: nn.Module, dtype=None) -> nn.Module:
+    """Fold every GBRA layer into a plain nn.Linear: W' = (m/n) * (W + s*B@A).
+
+    The merged model gives the same outputs as the GBRA model (using the current
+    buffered norm). 4-bit layers are dequantized, so the result is a dense model
+    you can export (e.g. to GGUF).
+    """
+    for module in list(model.modules()):
+        for name, child in list(module.named_children()):
+            if isinstance(child, GBRALinear):
+                W = base_weight(child.taban)
+                Wm = (child.m / child.n)[:, None] * (W + child.s * child.B.float() @ child.A.float())
+                bias = child.taban.bias
+                out_dtype = dtype or (bias.dtype if bias is not None else torch.float32)
+                lin = nn.Linear(Wm.shape[1], Wm.shape[0], bias=bias is not None,
+                                device=Wm.device, dtype=out_dtype)
+                lin.weight.copy_(Wm.to(out_dtype))
+                if bias is not None:
+                    lin.bias.copy_(bias.detach().to(out_dtype))
+                setattr(module, name, lin)
+    if hasattr(model, "_gbra_config"):
+        del model._gbra_config
+    return model
+
+
+# Turkish aliases
+gbra_ekle = apply_gbra
+tum_normlari_yenile = refresh_norms
+gbra_kaydet = save_gbra
+gbra_yukle = load_gbra
+gbra_birlestir = merge_gbra

gbra-0.1.1/src/gbra.egg-info/PKG-INFO

@@ -0,0 +1,108 @@
+Metadata-Version: 2.4
+Name: gbra
+Version: 0.1.1
+Summary: GBRA (Gember-Berke Rank Adaptation): DoRA-style fine-tuning with a buffered, lazily refreshed weight norm.
+Author: Berke T. Adanc
+License: MIT
+Keywords: lora,dora,peft,fine-tuning,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.0
+Requires-Dist: safetensors>=0.4
+Provides-Extra: hf
+Requires-Dist: transformers>=4.40; extra == "hf"
+Provides-Extra: 4bit
+Requires-Dist: bitsandbytes>=0.43; extra == "4bit"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: transformers>=4.40; extra == "test"
+Dynamic: license-file
+
+# GBRA: Gember-Berke Rank Adaptation
+
+*Pronounced "Gember RA". A GEMBER && Apex Logiç Studio project by Berke T. Adanç.*
+
+GBRA is a DoRA-style parameter-efficient fine-tuning method. It keeps DoRA's split of each weight into a learned **magnitude** and a low-rank-updated **direction**, but **buffers the weight norm** instead of recomputing it in every forward pass.
+
+```
+DoRA:  W' = m · (W + s·BA) / ‖W + s·BA‖     ← norm recomputed every forward (and again during gradient-checkpoint recompute)
+GBRA:  W' = m · (W + s·BA) / n               ← n is a stored buffer, refreshed every K optimizer steps
+```
+
+**Why it helps:** With gradient accumulation, the weights only change once per optimizer step. Recomputing the norm for every micro-batch repeats the same work many times. GBRA computes it once and reuses it.
+
+## Install
+
+```bash
+pip install gbra             # core
+pip install "gbra[hf]"       # + Hugging Face Trainer callback
+pip install "gbra[4bit]"     # + 4-bit (bitsandbytes) base models
+```
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, Trainer
+from gbra import apply_gbra, GBRACallback, save_gbra, merge_gbra
+
+model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
+model = apply_gbra(model, r=16, alpha=16)          # wraps q/k/v/o/gate/up/down projections
+
+trainer = Trainer(model=model, ..., callbacks=[GBRACallback(refresh_every=10)])
+trainer.train()
+
+save_gbra(model, "my_adapter")                     # small adapter (A, B, m, n)
+merge_gbra(model)                                  # fold into plain nn.Linear for export
+```
+
+Turkish aliases are also available: `gbra_ekle`, `tum_normlari_yenile`, `gbra_kaydet`, `gbra_yukle`, `gbra_birlestir`.
+
+## Preliminary results
+
+Qwen2.5-0.5B-Instruct, 3,000 Turkish chat/tool-use examples, 375 optimizer steps (batch 1 × 8 accumulation), r=16, single Kaggle T4, fp16 autocast. Same data, seed and hyper-parameters for every method. DoRA and LoRA via 🤗 PEFT.
+
+| Method | s / step | vs LoRA | Peak memory | Val. loss |
+|---|---|---|---|---|
+| LoRA | **5.77** | 1.00× | 8.11 GB | 1.6376 |
+| DoRA (PEFT) | 9.10 | 1.58× | 8.11 GB | 1.6285 |
+| **GBRA K=10** | 6.47 | 1.12× | 8.11 GB | **1.6249** |
+| GBRA K=100 | 6.47 | 1.12× | 8.11 GB | 1.6253 |
+| GBRA K=never | 6.46 | 1.12× | 8.11 GB | 1.6263 |
+
+**In this run, GBRA was ~29% faster than DoRA per step, with equal or slightly lower validation loss.** A shorter 125-step run gave the same speed ratio.
+
+### Limitations (please read)
+
+- **Single seed, single small model, single dataset.** The loss differences between GBRA variants and DoRA are small (≈0.001–0.004) and may be noise; repeated runs with more seeds are in progress.
+- Much of the speed-up comes from **not recomputing the norm per micro-batch**. With no gradient accumulation and no gradient checkpointing, the gap to DoRA should be smaller.
+- The comparison is against PEFT's DoRA implementation at the time of testing, not against every possible DoRA implementation.
+- 4-bit (bitsandbytes) support is implemented but **has not yet been benchmarked**.
+- Unlike PEFT LoRA, GBRA does not currently use fused kernels such as Unsloth's.
+
+## Credits
+
+GBRA builds directly on **DoRA** (Liu et al., 2024, *"DoRA: Weight-Decomposed Low-Rank Adaptation"*) and **LoRA** (Hu et al., 2021). Related ideas on learned scaling vectors appear in (IA)³ and VeRA. The contribution here is the buffered / lazily refreshed norm and its evaluation.
+
+The core `forward` of `GBRALinear` was written by Berke T. Adanç. Parts of the surrounding library and tests were developed with the help of an AI assistant.
+
+## Changelog
+
+- **0.1.1**: Python 3.9 compatibility fix in `GBRACallback`; new `refresh_on_end` option (refreshes the norm once more when training ends); README install instructions for PyPI.
+- **0.1.0**: First release.
+
+## License
+
+MIT
+
+---
+
+## Türkçe özet
+
+GBRA, DoRA'nın "büyüklük + yön" ayrımını korur, ama ağırlık normunu her adımda yeniden hesaplamak yerine **saklar** ve sadece **K optimizer adımında bir** günceller. Gradient birikimi kullanılan eğitimde ağırlıklar mini-batch'ler arasında değişmediği için DoRA aynı normu tekrar tekrar hesaplar. GBRA bu tekrarı ortadan kaldırır.
+
+İlk deneyde (Qwen2.5-0.5B, 375 adım, tek seed) GBRA, DoRA'dan **~%29 daha hızlı** ve doğrulama loss'u aynı ya da biraz daha iyiydi. Sonuçlar tek seed'e dayandığı için **ön sonuç** olarak değerlendirilmelidir.

gbra-0.1.1/src/gbra.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENSE
+README.md
+pyproject.toml
+src/gbra/__init__.py
+src/gbra/callback.py
+src/gbra/layer.py
+src/gbra/model.py
+src/gbra.egg-info/PKG-INFO
+src/gbra.egg-info/SOURCES.txt
+src/gbra.egg-info/dependency_links.txt
+src/gbra.egg-info/requires.txt
+src/gbra.egg-info/top_level.txt
+tests/test_gbra.py

gbra-0.1.1/src/gbra.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

gbra-0.1.1/src/gbra.egg-info/requires.txt

@@ -0,0 +1,12 @@
+torch>=2.0
+safetensors>=0.4
+
+[4bit]
+bitsandbytes>=0.43
+
+[hf]
+transformers>=4.40
+
+[test]
+pytest
+transformers>=4.40

gbra-0.1.1/src/gbra.egg-info/top_level.txt

@@ -0,0 +1 @@
+gbra

gbra-0.1.1/tests/test_gbra.py

@@ -0,0 +1,202 @@
+import torch
+import torch.nn as nn
+import pytest
+
+from gbra import (GBRALinear, apply_gbra, refresh_norms, save_gbra, load_gbra,
+                  merge_gbra, GBRACallback)
+
+TOL = 1e-5
+
+
+def dora_reference(layer: GBRALinear, x):
+    """DoRA formula: the norm is recomputed from scratch every time."""
+    W = layer.taban.weight
+    V = W + layer.s * layer.B @ layer.A
+    y = x @ (layer.m[:, None] * V / V.norm(dim=1)[:, None]).T
+    return y + layer.taban.bias if layer.taban.bias is not None else y
+
+
+@pytest.fixture
+def layer():
+    torch.manual_seed(0)
+    return GBRALinear(nn.Linear(32, 24), r=4, alpha=8)
+
+
+def perturb(layer):
+    with torch.no_grad():
+        layer.B.normal_(0, 0.1)
+        layer.m.mul_(1.3)
+
+
+def test_identity_at_init(layer):
+    x = torch.randn(5, 32)
+    assert torch.allclose(layer(x), layer.taban(x), atol=TOL)
+
+
+def test_equals_dora_after_refresh(layer):
+    perturb(layer)
+    layer.normu_yenile()
+    x = torch.randn(5, 32)
+    assert torch.allclose(layer(x), dora_reference(layer, x), atol=TOL)
+
+
+def test_uses_buffered_norm_before_refresh(layer):
+    perturb(layer)
+    layer.normu_yenile()
+    with torch.no_grad():
+        layer.B.add_(0.05)
+    x = torch.randn(5, 32)
+    W, b = layer.taban.weight, layer.taban.bias
+    expected = (layer.m / layer.n) * (x @ W.T + layer.s * (x @ layer.A.T) @ layer.B.T) + b
+    assert torch.allclose(layer(x), expected, atol=TOL)
+    assert not torch.allclose(layer(x), dora_reference(layer, x), atol=TOL)
+
+
+def test_no_bias_and_3d_input():
+    l = GBRALinear(nn.Linear(32, 24, bias=False), r=4)
+    x = torch.randn(2, 7, 32)
+    assert l(x).shape == (2, 7, 24)
+    assert torch.allclose(l(x), l.taban(x), atol=TOL)
+
+
+def test_gradients(layer):
+    perturb(layer)
+    layer(torch.randn(5, 32)).sum().backward()
+    for p in (layer.A, layer.B, layer.m):
+        assert p.grad is not None and p.grad.abs().sum() > 0
+    assert layer.taban.weight.grad is None
+    assert not layer.n.requires_grad
+
+
+def test_magnitude_scales_output(layer):
+    """Berke'nin testi: m iki katina cikinca (bias'siz) cikti da iki katina cikmali."""
+    l = GBRALinear(nn.Linear(32, 24, bias=False), r=4)
+    x = torch.randn(3, 32)
+    y1 = l(x)
+    with torch.no_grad():
+        l.m.mul_(2)
+    assert torch.allclose(l(x), 2 * y1, atol=TOL)
+
+
+def test_rejects_bad_rank():
+    with pytest.raises(ValueError):
+        GBRALinear(nn.Linear(4, 4), r=0)
+
+
+# ---------------- model-level ----------------
+
+class Tiny(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.q_proj = nn.Linear(16, 16)
+        self.up_proj = nn.Linear(16, 32, bias=False)
+        self.other = nn.Linear(32, 8)
+
+    def forward(self, x):
+        return self.other(torch.relu(self.up_proj(self.q_proj(x))))
+
+
+def trained_tiny():
+    torch.manual_seed(1)
+    m = apply_gbra(Tiny(), targets=("q_proj", "up_proj"), r=4, alpha=4, verbose=False)
+    for mod in m.modules():
+        if isinstance(mod, GBRALinear):
+            perturb(mod)
+    refresh_norms(m)
+    return m
+
+
+def test_apply_only_targets_and_freezes():
+    m = apply_gbra(Tiny(), targets=("q_proj", "up_proj"), verbose=False)
+    assert isinstance(m.q_proj, GBRALinear) and isinstance(m.up_proj, GBRALinear)
+    assert isinstance(m.other, nn.Linear) and not isinstance(m.other, GBRALinear)
+    assert not any(p.requires_grad for p in m.other.parameters())
+
+
+def test_apply_raises_when_nothing_matches():
+    with pytest.raises(ValueError):
+        apply_gbra(Tiny(), targets=("nope",), verbose=False)
+
+
+def test_refresh_norms_count():
+    assert refresh_norms(trained_tiny()) == 2
+
+
+def test_merge_matches_gbra_output():
+    m = trained_tiny()
+    x = torch.randn(4, 16)
+    before = m(x)
+    merge_gbra(m)
+    assert not any(isinstance(mod, GBRALinear) for mod in m.modules())
+    assert torch.allclose(m(x), before, atol=1e-5)
+
+
+def test_save_load_roundtrip(tmp_path):
+    m = trained_tiny()
+    x = torch.randn(4, 16)
+    expected = m(x)
+    save_gbra(m, tmp_path)
+    torch.manual_seed(1)
+    fresh = Tiny()
+    # same base weights as the trained model's base layers
+    fresh.q_proj.load_state_dict(m.q_proj.taban.state_dict())
+    fresh.up_proj.load_state_dict(m.up_proj.taban.state_dict())
+    fresh.other.load_state_dict(m.other.state_dict())
+    load_gbra(fresh, tmp_path)
+    assert torch.allclose(fresh(x), expected, atol=TOL)
+
+
+def test_callback_refreshes_every_k():
+    m = trained_tiny()
+    with torch.no_grad():
+        m.q_proj.B.add_(0.1)
+    old = m.q_proj.n.clone()
+
+    class S:  # minimal TrainerState stand-in
+        global_step = 3
+    cb = GBRACallback(refresh_every=5)
+    cb.on_step_end(None, S, None, model=m)
+    assert torch.equal(m.q_proj.n, old)          # 3 % 5 != 0 -> no refresh
+    S.global_step = 5
+    cb.on_step_end(None, S, None, model=m)
+    assert not torch.equal(m.q_proj.n, old)      # refreshed
+
+
+def test_qwen2_tiny_trains():
+    transformers = pytest.importorskip("transformers")
+    cfg = transformers.Qwen2Config(vocab_size=200, hidden_size=64, intermediate_size=128,
+                                   num_hidden_layers=2, num_attention_heads=4, num_key_value_heads=2)
+    torch.manual_seed(0)
+    model = transformers.Qwen2ForCausalLM(cfg)
+    model.gradient_checkpointing_enable()
+    model.enable_input_require_grads()
+    model.config.use_cache = False
+    apply_gbra(model, r=4, alpha=4, verbose=False)
+    x = torch.randint(0, 200, (1, 32))
+    opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-2)
+    first = model(input_ids=x, labels=x).loss.item()
+    for _ in range(10):
+        loss = model(input_ids=x, labels=x).loss
+        loss.backward(); opt.step(); opt.zero_grad()
+        refresh_norms(model)
+    assert loss.item() < first
+
+
+def test_callback_refresh_on_end():
+    m = trained_tiny()
+    with torch.no_grad():
+        m.q_proj.B.add_(0.1)
+    old = m.q_proj.n.clone()
+    GBRACallback(refresh_every=None).on_train_end(None, None, None, model=m)
+    assert not torch.equal(m.q_proj.n, old)
+    m2 = trained_tiny()
+    with torch.no_grad():
+        m2.q_proj.B.add_(0.1)
+    old2 = m2.q_proj.n.clone()
+    GBRACallback(refresh_every=None, refresh_on_end=False).on_train_end(None, None, None, model=m2)
+    assert torch.equal(m2.q_proj.n, old2)
+
+
+def test_version():
+    import gbra
+    assert gbra.__version__ == "0.1.1"