aria-trl 1.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,146 @@
1
+ # Byte-compiled / optimized / DLL files
2
+ __pycache__/
3
+ *.py[cod]
4
+ *$py.class
5
+
6
+ # C extensions
7
+ *.so
8
+
9
+ # Distribution / packaging
10
+ .Python
11
+ build/
12
+ develop-eggs/
13
+ dist/
14
+ downloads/
15
+ eggs/
16
+ .eggs/
17
+ lib/
18
+ lib64/
19
+ parts/
20
+ sdist/
21
+ var/
22
+ wheels/
23
+ pip-wheel-metadata/
24
+ share/python-wheels/
25
+ *.egg-info/
26
+ .installed.cfg
27
+ *.egg
28
+ MANIFEST
29
+
30
+ # PyInstaller
31
+ *.manifest
32
+ *.spec
33
+
34
+ # Installer logs
35
+ pip-log.txt
36
+ pip-delete-this-directory.txt
37
+
38
+ # Unit test / coverage reports
39
+ htmlcov/
40
+ .tox/
41
+ .nox/
42
+ .coverage
43
+ .coverage.*
44
+ .cache
45
+ nosetests.xml
46
+ coverage.xml
47
+ *.cover
48
+ *.py,cover
49
+ .hypothesis/
50
+ .pytest_cache/
51
+
52
+ # Translations
53
+ *.mo
54
+ *.pot
55
+
56
+ # Django stuff:
57
+ *.log
58
+ local_settings.py
59
+ db.sqlite3
60
+ db.sqlite3-journal
61
+
62
+ # Flask stuff:
63
+ instance/
64
+ .webassets-cache
65
+
66
+ # Scrapy stuff:
67
+ .scrapy
68
+
69
+ # Sphinx documentation
70
+ docs/_build/
71
+
72
+ # PyBuilder
73
+ target/
74
+
75
+ # Jupyter Notebook
76
+ .ipynb_checkpoints
77
+
78
+ # IPython
79
+ profile_default/
80
+ ipython_config.py
81
+
82
+ # pyenv
83
+ .python-version
84
+
85
+ # pipenv
86
+ Pipfile.lock
87
+
88
+ # PEP 582
89
+ __pypackages__/
90
+
91
+ # Celery stuff
92
+ celerybeat-schedule
93
+ celerybeat.pid
94
+
95
+ # SageMath parsed files
96
+ *.sage.py
97
+
98
+ # Environments
99
+ .env
100
+ .venv
101
+ env/
102
+ venv/
103
+ ENV/
104
+ env.bak/
105
+ venv.bak/
106
+
107
+ # Spyder project settings
108
+ .spyderproject
109
+ .spyproject
110
+
111
+ # Rope project settings
112
+ .ropeproject
113
+
114
+ # mkdocs documentation
115
+ /site
116
+
117
+ # mypy
118
+ .mypy_cache/
119
+ .dmypy.json
120
+ dmypy.json
121
+
122
+ # Pyre type checker
123
+ .pyre/
124
+
125
+ # IDE
126
+ .vscode/
127
+ .idea/
128
+ *.swp
129
+ *.swo
130
+ *~
131
+ .DS_Store
132
+
133
+ # Model checkpoints and results
134
+ checkpoints/
135
+ results/
136
+ *.pt
137
+ *.pth
138
+ *.bin
139
+
140
+ # Tensorboard
141
+ runs/
142
+ logs/
143
+
144
+ # Temporary files
145
+ *.tmp
146
+ *.temp
aria_trl-1.0.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Darshan Poudel
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,301 @@
1
+ Metadata-Version: 2.4
2
+ Name: aria-trl
3
+ Version: 1.0.0
4
+ Summary: Continual learning for LLM fine-tuning via ARIA mechanisms (PlasticityGatedMLP, SPC, Task Adapters)
5
+ Home-page: https://github.com/rsd-darshan/aria-trl
6
+ Author: Darshan Poudel
7
+ Author-email: Darshan Poudel <poudeldarshan44@gmail.com>
8
+ License: MIT
9
+ Project-URL: Homepage, https://github.com/rsd-darshan/aria-trl
10
+ Project-URL: Repository, https://github.com/rsd-darshan/aria-trl.git
11
+ Project-URL: Issues, https://github.com/rsd-darshan/aria-trl/issues
12
+ Keywords: continual-learning,llm,fine-tuning,catastrophic-forgetting,fast-slow-pathways,fisher-consolidation
13
+ Classifier: Development Status :: 4 - Beta
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: Intended Audience :: Science/Research
16
+ Classifier: License :: OSI Approved :: MIT License
17
+ Classifier: Operating System :: OS Independent
18
+ Classifier: Programming Language :: Python :: 3
19
+ Classifier: Programming Language :: Python :: 3.10
20
+ Classifier: Programming Language :: Python :: 3.11
21
+ Classifier: Programming Language :: Python :: 3.12
22
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
23
+ Requires-Python: >=3.10
24
+ Description-Content-Type: text/markdown
25
+ License-File: LICENSE
26
+ Requires-Dist: torch>=2.0.0
27
+ Requires-Dist: transformers>=4.30.0
28
+ Requires-Dist: trl>=0.7.0
29
+ Requires-Dist: datasets>=2.10.0
30
+ Requires-Dist: numpy>=1.23.0
31
+ Provides-Extra: dev
32
+ Requires-Dist: pytest>=7.0; extra == "dev"
33
+ Requires-Dist: black>=23.0; extra == "dev"
34
+ Requires-Dist: isort>=5.12; extra == "dev"
35
+ Requires-Dist: mypy>=1.0; extra == "dev"
36
+ Provides-Extra: examples
37
+ Requires-Dist: wandb>=0.15.0; extra == "examples"
38
+ Dynamic: author
39
+ Dynamic: home-page
40
+ Dynamic: license-file
41
+ Dynamic: requires-python
42
+
43
+ # aria-trl: Continual Learning for LLM Fine-tuning
44
+
45
+ [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
46
+ [![PyTorch 2.0+](https://img.shields.io/badge/PyTorch-2.0%2B-ee4c2c.svg)](https://pytorch.org/)
47
+ [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
48
+
49
+ **aria-trl** brings ARIA's continual learning mechanisms to Hugging Face's TRL library, enabling fine-tuning of large language models on sequential tasks without catastrophic forgetting.
50
+
51
+ ## Overview
52
+
53
+ Fine-tuning LLMs sequentially on multiple tasks typically leads to **catastrophic forgetting** — the model forgets earlier tasks while learning new ones. aria-trl prevents this through three ARIA mechanisms:
54
+
55
+ 1. **PlasticityGatedMLP**: Dual fast/slow pathways in FFN layers
56
+ - Fast pathway: volatile, learns task-specific patterns
57
+ - Slow pathway: stable, retains task-generic knowledge
58
+ - Learned gate routes computation per token
59
+
60
+ 2. **Slow-Pathway Consolidation (SPC)**: Fisher-weighted regularization
61
+ - Estimates diagonal Fisher Information after each task
62
+ - Protects consolidated knowledge from being overwritten
63
+ - 50% fewer parameters than standard EWC
64
+
65
+ 3. **Task-Specific Adapters**: Lightweight LoRA-like modules
66
+ - One residual adapter per task
67
+ - Frozen after training to prevent overwriting
68
+ - Minimal parameter overhead
69
+
70
+ ## Features
71
+
72
+ - ✅ **Drop-in SFTTrainer subclass** — minimal code changes
73
+ - ✅ **Asymmetric learning rates** — slow pathway learns slower
74
+ - ✅ **Gradient dampening** — slow grads scaled by (1−π̄)
75
+ - ✅ **Works with any HF model** — LLaMA, Mistral, DistilGPT2, etc.
76
+ - ✅ **Full checkpoint support** — save/load with consolidation state
77
+ - ✅ **Production-ready** — comprehensive error handling
78
+
79
+ ## Installation
80
+
81
+ ```bash
82
+ pip install aria-trl
83
+ ```
84
+
85
+ Or install from source:
86
+
87
+ ```bash
88
+ git clone https://github.com/rsd-darshan/aria-trl.git
89
+ cd aria-trl
90
+ pip install -e .
91
+ ```
92
+
93
+ ## Quick Start
94
+
95
+ ```python
96
+ from aria_trl import ContinualSFTTrainer, ARIAConfig
97
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments
98
+
99
+ # Load model
100
+ model = AutoModelForSequenceClassification.from_pretrained("distilgpt2", num_labels=2)
101
+ tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
102
+
103
+ # Configure ARIA
104
+ aria_config = ARIAConfig(
105
+ plasticity_lambda=0.01, # bimodal gate specialization
106
+ spc_lambda=100.0, # Fisher consolidation strength
107
+ adapter_dim=64, # task adapter bottleneck
108
+ slow_lr_ratio=0.5, # asymmetric LR: slow=0.5x fast
109
+ )
110
+
111
+ # Training arguments
112
+ args = TrainingArguments(
113
+ output_dir="./checkpoints",
114
+ learning_rate=2e-4,
115
+ num_train_epochs=3,
116
+ per_device_train_batch_size=8,
117
+ )
118
+
119
+ # Create trainer
120
+ trainer = ContinualSFTTrainer(
121
+ model=model,
122
+ tokenizer=tokenizer,
123
+ args=args,
124
+ train_dataset=task1_train,
125
+ eval_dataset=task1_eval,
126
+ aria_config=aria_config,
127
+ )
128
+
129
+ # Train on task 1
130
+ trainer.train()
131
+
132
+ # Consolidate before next task (Fisher estimation)
133
+ trainer.consolidate_task(task_id=0)
134
+
135
+ # Train on task 2
136
+ trainer.add_task(task_id=1)
137
+ trainer.train_dataset = task2_train
138
+ trainer.eval_dataset = task2_eval
139
+ trainer.train()
140
+
141
+ # Consolidate again
142
+ trainer.consolidate_task(task_id=1)
143
+
144
+ # Evaluate on all tasks to measure forgetting
145
+ metrics = trainer.evaluate_all_tasks(all_tasks)
146
+ ```
147
+
148
+ ## Core Components
149
+
150
+ ### ARIAConfig
151
+
152
+ Configuration for ARIA continual learning:
153
+
154
+ ```python
155
+ from aria_trl import ARIAConfig
156
+
157
+ config = ARIAConfig(
158
+ plasticity_lambda=0.01, # Bimodal specialization loss weight
159
+ spc_lambda=100.0, # Fisher regularization strength
160
+ adapter_dim=64, # Task adapter bottleneck dimension
161
+ slow_lr_ratio=0.5, # Slow pathway LR multiplier
162
+ warmup_steps=500, # Before plasticity loss activates
163
+ consolidation_steps_per_task=None # Fisher estimation steps (None=all)
164
+ )
165
+ ```
166
+
167
+ ### PlasticityGatedMLP
168
+
169
+ Replaces transformer FFN layers automatically. Routes per-token computation:
170
+
171
+ ```
172
+ output = π * fast_pathway(x) + (1−π) * slow_pathway(x)
173
+ ```
174
+
175
+ where π ∈ (0,1) is learned and regularized to specialize (0 or 1).
176
+
177
+ ### TaskFastAdapter
178
+
179
+ Per-task residual adapter (LoRA-like):
180
+
181
+ ```
182
+ h_new = h + adapter(h)
183
+ ```
184
+
185
+ Bottleneck design: `h → compress → ReLU → expand`, zero-initialized expansion.
186
+
187
+ ### FisherConsolidator
188
+
189
+ Estimates diagonal Fisher Information on validation data after each task:
190
+
191
+ ```python
192
+ consolidator = FisherConsolidator(model, device)
193
+ consolidator.consolidate(task_id, eval_loader)
194
+ loss = consolidator.compute_spc_loss(global_step)
195
+ ```
196
+
197
+ ## Metrics
198
+
199
+ Compute continual learning metrics with:
200
+
201
+ ```python
202
+ from aria_trl.utils import compute_continual_metrics
203
+
204
+ metrics = compute_continual_metrics(task_accuracies)
205
+ # Returns: avg_accuracy, forgetting, forward_transfer
206
+ ```
207
+
208
+ - **Average Accuracy**: Mean accuracy on all tasks at end
209
+ - **Forgetting**: How much old tasks degrade (lower is better)
210
+ - **Forward Transfer**: How much new tasks benefit from old tasks
211
+
212
+ ## Example: DistilGPT2 on Sequential Tasks
213
+
214
+ See `examples/distilgpt2_example.py` for a complete working example:
215
+
216
+ ```bash
217
+ python examples/distilgpt2_example.py
218
+ ```
219
+
220
+ Trains on 3 sequential tasks (sentiment, toxicity, spam) and prints metrics.
221
+
222
+ ## How ARIA Prevents Forgetting
223
+
224
+ ### The Problem
225
+ Standard fine-tuning on sequential tasks overwrites weights learned on earlier tasks:
226
+
227
+ ```
228
+ Task 1: [update all weights]
229
+ Task 2: [update all weights again] ← Task 1 weights overwritten
230
+ Task 3: [update all weights again] ← Task 1 & 2 weights overwritten
231
+ Eval: [accuracy on Task 1 ↓↓↓] ← Catastrophic forgetting!
232
+ ```
233
+
234
+ ### The Solution
235
+ ARIA separates fast (volatile) and slow (stable) pathways:
236
+
237
+ ```
238
+ Fast pathway: [updates freely, learns task-specific patterns]
239
+ Slow pathway: [updates slowly, consolidated by Fisher regularization]
240
+ Gate π: [routes each token: fast for new patterns, slow for stability]
241
+
242
+ Result: Old tasks remain in slow pathway, new tasks learn in fast pathway
243
+ ```
244
+
245
+ ## Design Philosophy
246
+
247
+ aria-trl's design prioritizes:
248
+
249
+ 1. **Compatibility**: Subclasses SFTTrainer, works with any HF model
250
+ 2. **Simplicity**: Three mechanisms, no complex multi-model ensembles
251
+ 3. **Efficiency**: Task adapters add <1% parameters, Fisher diagonal reduces memory
252
+ 4. **Interpretability**: Gate π shows per-token plasticity, easy to debug
253
+
254
+ ## Papers & Motivation
255
+
256
+ - **ARIA** (PyTorch): ["Adaptive Recurrent Intelligence Architecture"](https://github.com/rsd-darshan/ARIA) — core continual learning research, CNNs and task-incremental learning
257
+ - **aria-trl** (TRL): This package — brings ARIA mechanisms to LLM fine-tuning via Hugging Face
258
+ - **NCG**: ["Novelty-triggered Capacity Growth"](https://github.com/rsd-darshan/NCG) — reactive capacity expansion (predecessor)
259
+ - **EWC**: ["Elastic Weight Consolidation"](https://arxiv.org/abs/1612.00796) — standard baseline for catastrophic forgetting
260
+
261
+ ## Limitations & Future Work
262
+
263
+ - **Slow pathway at 0.5× LR**: Conservative; tuning improves CIFAR-10 accuracy
264
+ - **Fisher diagonal only**: Full Fisher or Kronecker-factored approximations are future work
265
+ - **DistilGPT2 tested**: Larger models (7B+) need memory optimization
266
+ - **Binary classification**: Multi-class and language modeling tasks need validation
267
+
268
+ ## Contributing
269
+
270
+ Contributions welcome! Please:
271
+
272
+ 1. Fork the repo
273
+ 2. Create a feature branch (`git checkout -b feature/my-feature`)
274
+ 3. Commit changes (no AI attribution required)
275
+ 4. Push to remote and open a PR
276
+
277
+ ## License
278
+
279
+ MIT License — see [LICENSE](LICENSE) for details.
280
+
281
+ ## Citation
282
+
283
+ If you use aria-trl in your research, please cite:
284
+
285
+ ```bibtex
286
+ @software{poudel2026aria_trl,
287
+ title = {aria-trl: Continual Learning for LLM Fine-tuning},
288
+ author = {Poudel, Darshan},
289
+ year = {2026},
290
+ url = {https://github.com/rsd-darshan/aria-trl}
291
+ }
292
+ ```
293
+
294
+ ## Contact
295
+
296
+ - **GitHub**: [@rsd-darshan](https://github.com/rsd-darshan)
297
+ - **Email**: poudeldarshan44@gmail.com
298
+
299
+ ---
300
+
301
+ **aria-trl** — continual learning, without the catastrophe.