trainers-rb 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +40 -0
- data/README.md +91 -34
- data/lib/trainers/data/data_collator.rb +20 -24
- data/lib/trainers/data/dataset.rb +149 -0
- data/lib/trainers/lora/lora_config.rb +32 -1
- data/lib/trainers/lora/lora_model.rb +7 -4
- data/lib/trainers/lora/lora_utils.rb +25 -12
- data/lib/trainers/trainer.rb +120 -38
- data/lib/trainers/training_arguments.rb +7 -1
- data/lib/trainers/version.rb +1 -1
- data/lib/trainers.rb +34 -2
- metadata +15 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 63872fbc41baf9c23b43cb48722c0f5a94becef7480c2452e5a51ba7b10ac817
|
|
4
|
+
data.tar.gz: 99cbdb6695ae3952cdcf37e991cb5d9e1471b49fa3d5346bc2a26bc88575354d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: '080c41aa2ba050105595f0ce4fadcc305daba493dae923bebe9e5a8610ab8c6138d48cafb34cd7445eb43b88663f1f45bf66a5c64043d70eb487d34812b34073'
|
|
7
|
+
data.tar.gz: 1d94c9069209dfa6a64d418e9d26fb7baa9329ac83514ab54848731dc6cd263415a903192e740c6820d440b5d5571be060939c5f87285c9ee2ac17ddda4b072a
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,42 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.4.0 (2026-11-10)
|
|
4
|
+
|
|
5
|
+
**Upgrade from 0.3.0 if you train on MPS or CUDA.** 0.3.0 crashes when `evaluate` runs before `train`, and can run out of GPU memory during evaluation. Both are fixed here (see Fixed).
|
|
6
|
+
|
|
7
|
+
**This release changes LoRA defaults.** `LoraConfig#target_modules` now defaults to the query and value layers of the model's architecture, and targets match by name parts rather than substrings. See Changed.
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- **Datasets from files** (#30): `Dataset.from_jsonl(path)` and `Dataset.from_csv(path, **csv_options)`. A malformed JSONL line raises `ConfigurationError` naming the file and line.
|
|
12
|
+
- **`Dataset#split(test_size:, seed: 42)`** returns `[train, test]`, shuffled with `Random.new(seed)`. `test_size` is a fraction or a row count.
|
|
13
|
+
- **`Dataset#map` and `Dataset#filter`** return new Datasets, and `map` passes the block a copy of each row, so `row.delete(:label)` leaves the source Dataset unchanged. `map` returns an Array, as before, when the block doesn't return Hashes. `select` still returns an Array.
|
|
14
|
+
- **`Dataset#tokenize(tokenizer, column: :text, max_length: 512, keep: false)`** (#31) adds `input_ids` and `attention_mask` (and `token_type_ids` for text pairs), truncated to `max_length`. It takes a transformers-rb tokenizer or a `Tokenizers::Tokenizer`, and restores the latter's truncation setting afterwards.
|
|
15
|
+
- **`Trainers.from_pretrained(labels:)`** (#32) sets `num_labels`, `id2label` and `label2id`, so `config.json`, `load_pretrained` and Python see the class names. A `num_labels:` that contradicts `labels:` raises `ArgumentError`.
|
|
16
|
+
- **A log line when training starts** (#35) with the device, dtype, number of examples, epochs, optimizer steps, batch and accumulation, and how many parameters are trainable.
|
|
17
|
+
- `from_pretrained(task: :question_answering)` warns that transformers-rb 0.2.0 can't compute the question-answering loss.
|
|
18
|
+
- **`max_steps:`** (#34) trains for that many optimizer steps, overriding `num_train_epochs`. Epochs continue until the budget is spent, including when callbacks end some early with `should_epoch_stop`. The learning-rate schedule and `warmup_ratio` use it as the total.
|
|
19
|
+
|
|
20
|
+
### Fixed
|
|
21
|
+
|
|
22
|
+
- **Token classification evaluates and predicts** (#27). `evaluate` crashed on per-token logits `[batch, seq, labels]`, and `predict` crashed when batches were padded to different lengths. Both now pad per-batch results to the longest batch with `-100` before joining them, so `compute_metrics` gets predictions and labels of matching shape. Training BERT for token classification also failed: transformers-rb 0.2.0's `BertForTokenClassification` raises `NameError` (`CrossEntropyLoss`) whenever it gets labels. The Trainer now runs it without labels and computes the loss itself.
|
|
23
|
+
- **Models whose `forward` takes no label keyword train and evaluate again.** `evaluate` passed the labels in and raised `ArgumentError: unknown keyword: :labels`. The Trainer now retries without them and computes the loss itself. Training gets the same fallback.
|
|
24
|
+
- **`evaluate` uses the model's own loss** when it returns one, instead of always recomputing cross-entropy over the logits. The fallback cross-entropy flattens per-token logits and ignores `-100` labels.
|
|
25
|
+
- **`eval_loss` is averaged over rows, not batches.** A short last batch used to count as much as a full one, so `eval_loss` can differ slightly from 0.3.0 when the eval set isn't a multiple of `per_device_eval_batch_size`.
|
|
26
|
+
- **CUDA is used when available** (#28). `resolved_device` only ever chose MPS or the CPU. The order is now an explicit `device:`, then CUDA, then MPS, then the CPU.
|
|
27
|
+
- **`DataCollatorWithPadding` truncates rows longer than `max_length`** (#29). Rows given as Arrays raised `ArgumentError: negative argument`. Rows given as Tensors were already truncated.
|
|
28
|
+
- **`DataCollatorWithPadding` pads Tensor labels with `-100`**, the same as Array labels. They used to be padded with `0`, a real class, so padding counted towards the loss.
|
|
29
|
+
- **Memory stays flat during training and evaluation.** torch-rb tensors are freed only when Ruby garbage-collects them, and Ruby can't see the memory LibTorch holds, so it rarely collects. Activations piled up batch after batch: evaluating DistilBERT on MPS ran out of memory at 22.6 GiB within 17 batches. The Trainer now runs a minor garbage collection after each batch, which kept that evaluation at 0.8 GB and costs under a millisecond per batch.
|
|
30
|
+
- **`evaluate` and `predict` move the model to the training device.** Only `train` did, so calling `evaluate` before `train` on MPS or CUDA, for a baseline, crashed with `Passed CPU tensor to MPS op`. Found by fine-tuning DistilBERT on MPS.
|
|
31
|
+
- `predict` no longer passes labels to the model, and puts the model back in training mode afterwards, as `evaluate` does.
|
|
32
|
+
|
|
33
|
+
### Changed
|
|
34
|
+
|
|
35
|
+
- **LoRA `target_modules` defaults per architecture** (#33). It was `["query", "value"]`, which matched nothing on DistilBERT. The default is now `nil`, meaning `q_lin`/`v_lin` for DistilBERT, `query`/`value` for BERT and XLM-RoBERTa, `query_proj`/`value_proj` for DeBERTa-v2 and `q`/`v` for MPNet. Other transformers models need `target_modules:`, and models without a config keep `["query", "value"]`.
|
|
36
|
+
- **LoRA targets match whole name parts**, as in PEFT. `"query"` matches `attention.self.query` but no longer `pos_query_proj`. A target that only matches by substring still works, with a deprecation warning; substring matching ends in 0.5.0.
|
|
37
|
+
- **README "Support" section** (#35) replaces "Supported Tasks" and "Device Support". It has a table of architectures and tasks that the test suite checks against transformers-rb, and a platform matrix that separates tested in CI, hand-tested and unsupported. Question answering is now listed as not trainable. It was listed as supported before.
|
|
38
|
+
- New runtime dependency: `csv`, which is no longer a default gem from Ruby 3.4.
|
|
39
|
+
|
|
3
40
|
## 0.3.0 (2026-11-10)
|
|
4
41
|
|
|
5
42
|
**This release changes LoRA training results.** `LoraModel.apply` now keeps the task head trainable, as PEFT does. It used to freeze it at its random initial weights. Expect better LoRA models than 0.2.0 with the same arguments. Full fine-tunes are unchanged.
|
|
@@ -19,6 +56,7 @@
|
|
|
19
56
|
- **`save_model` writes the tokenizer for transformers-rb tokenizers** (#17). It used to save `tokenizer.json` only for objects that respond to `save`, which ruled out `AutoTokenizer` results such as `DistilBertTokenizerFast`, so no tokenizer was ever written. It now saves the backend as `tokenizer.json`, plus `tokenizer_config.json` with the tokenizer class and special tokens.
|
|
20
57
|
- `model.safetensors` and `adapter_model.safetensors` carry the `"format" => "pt"` metadata that the transformers loaders in Ruby and Python check for.
|
|
21
58
|
|
|
59
|
+
|
|
22
60
|
### Changed
|
|
23
61
|
|
|
24
62
|
- README: the LoRA example uses DistilBERT's module names (`q_lin`, `v_lin`) and shows its real output. The roadmap lists what is not planned, and "Known limitations" is gone, because its three issues are fixed.
|
|
@@ -27,6 +65,7 @@
|
|
|
27
65
|
|
|
28
66
|
**This release changes training results.** Runs that use `gradient_accumulation_steps > 1` now follow the intended learning-rate schedule. All runs now apply `seed:`, so data order is different from 0.1.x and reproducible. Expect different (better) final models than 0.1.x with the same arguments.
|
|
29
67
|
|
|
68
|
+
|
|
30
69
|
### Changed
|
|
31
70
|
|
|
32
71
|
- **A step is now one optimizer update** (#12). `global_step`, `max_steps`, `logging_steps`, `eval_steps`, `save_steps`, `warmup_steps` and the scheduler all count optimizer updates, not batches. The scheduler is sized as `ceil(batches_per_epoch / gradient_accumulation_steps) * num_train_epochs`, so it now decays to the end. Previously, with accumulation 4, a linear schedule ended near 75% of the peak learning rate.
|
|
@@ -58,6 +97,7 @@
|
|
|
58
97
|
- `LoraModel.merge` no longer applies adapters twice. Merged layers kept adding the LoRA path on top of the merged weights, so a merged model's outputs were wrong. Each `LoraLinear` is now replaced by a plain `Torch::NN::Linear` holding the merged weights, so a saved merged model has the base model's parameter names. `LoraLinear#merge!` is idempotent and adds `#merged?` and `#to_linear`.
|
|
59
98
|
- `Trainers::VERSION` now matches the gem version. The gemspec reads it from `lib/trainers/version.rb`. (#2)
|
|
60
99
|
|
|
100
|
+
|
|
61
101
|
### Changed
|
|
62
102
|
|
|
63
103
|
- `TrainingArguments` validates its values at construction, and raises `ArgumentError` naming the argument, the value received and what was expected (#8):
|
data/README.md
CHANGED
|
@@ -42,23 +42,19 @@ For Linux, and for running the tests, see [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
|
42
42
|
```ruby
|
|
43
43
|
require "trainers-rb"
|
|
44
44
|
|
|
45
|
-
# Load a pre-trained model and tokenizer
|
|
45
|
+
# Load a pre-trained model and tokenizer. labels: names the classes in id
|
|
46
|
+
# order; they are saved in config.json.
|
|
46
47
|
model, tokenizer = Trainers.from_pretrained(
|
|
47
48
|
"distilbert-base-uncased",
|
|
48
49
|
task: :sequence_classification,
|
|
49
|
-
|
|
50
|
+
labels: %w[negative positive]
|
|
50
51
|
)
|
|
51
52
|
|
|
52
|
-
#
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
attention_mask: encoded["attention_mask"],
|
|
58
|
-
labels: labels[i]
|
|
59
|
-
}
|
|
60
|
-
end
|
|
61
|
-
train_dataset = Trainers::Dataset.new(train_data)
|
|
53
|
+
# reviews.jsonl: {"text": "A gripping film.", "label": 1} on each line
|
|
54
|
+
dataset = Trainers::Dataset.from_jsonl("reviews.jsonl")
|
|
55
|
+
.map { |row| row.merge(labels: row.delete(:label)) }
|
|
56
|
+
.tokenize(tokenizer, column: :text, max_length: 128)
|
|
57
|
+
train_dataset, val_dataset = dataset.split(test_size: 0.1, seed: 42)
|
|
62
58
|
|
|
63
59
|
# Configure and train
|
|
64
60
|
args = Trainers::TrainingArguments.new(
|
|
@@ -122,7 +118,7 @@ config = Trainers::LoraConfig.new(
|
|
|
122
118
|
r: 8, # rank
|
|
123
119
|
lora_alpha: 16, # scaling factor
|
|
124
120
|
lora_dropout: 0.1,
|
|
125
|
-
target_modules: ["q_lin", "v_lin"], #
|
|
121
|
+
# target_modules: ["q_lin", "v_lin"], # defaults to the attention query and value layers
|
|
126
122
|
bias: :none, # :none, :all, or :lora_only
|
|
127
123
|
task_type: :sequence_classification
|
|
128
124
|
# modules_to_save: ["classifier"] # defaults to the task head for task_type
|
|
@@ -130,7 +126,7 @@ config = Trainers::LoraConfig.new(
|
|
|
130
126
|
|
|
131
127
|
Trainers::LoraModel.apply(model, config)
|
|
132
128
|
# => Training in full: pre_classifier, classifier
|
|
133
|
-
# => LoRA applied to 12 modules: ...
|
|
129
|
+
# => LoRA target_modules ["q_lin", "v_lin"] (default) applied to 12 modules: ...
|
|
134
130
|
# => trainable params: 739,586 || all params: 67,102,466 || trainable%: 1.1022%
|
|
135
131
|
|
|
136
132
|
# Train as usual
|
|
@@ -154,6 +150,19 @@ The task head is trained in full, as in PEFT. `from_pretrained` initialises it r
|
|
|
154
150
|
|
|
155
151
|
Names match the last part of a module's name. Pass `modules_to_save: []` to freeze the head, or your own list for other models.
|
|
156
152
|
|
|
153
|
+
`target_modules` defaults to the attention query and value layers of the model's `config.model_type`:
|
|
154
|
+
|
|
155
|
+
| `model_type` | Default `target_modules` |
|
|
156
|
+
|---|---|
|
|
157
|
+
| `bert`, `xlm-roberta` | `query`, `value` |
|
|
158
|
+
| `distilbert` | `q_lin`, `v_lin` |
|
|
159
|
+
| `deberta-v2` | `query_proj`, `value_proj` |
|
|
160
|
+
| `mpnet` | `q`, `v` |
|
|
161
|
+
|
|
162
|
+
Any other `model_type` needs `target_modules:`. Models without a transformers config default to `query` and `value`. Pass `:all_linear` to adapt every Linear layer.
|
|
163
|
+
|
|
164
|
+
A target matches a module when it is the module's full name or the last part of it, as in PEFT: `"query"` matches `attention.self.query` but not `pos_query_proj`. A target that matches nothing that way still matches by substring, with a deprecation warning. Substring matching ends in 0.5.0.
|
|
165
|
+
|
|
157
166
|
### Loading saved LoRA adapters
|
|
158
167
|
|
|
159
168
|
```ruby
|
|
@@ -195,6 +204,7 @@ Resume with the same `seed`, dataset and arguments as the original run. The trai
|
|
|
195
204
|
|----------|---------|-------------|
|
|
196
205
|
| `output_dir` | `"./output"` | Directory for checkpoints and saved models |
|
|
197
206
|
| `num_train_epochs` | `3` | Number of training epochs |
|
|
207
|
+
| `max_steps` | `nil` | Number of optimizer steps to train for. Overrides `num_train_epochs` when set |
|
|
198
208
|
| `per_device_train_batch_size` | `8` | Training batch size |
|
|
199
209
|
| `per_device_eval_batch_size` | `8` | Evaluation batch size |
|
|
200
210
|
| `learning_rate` | `5e-5` | Peak learning rate for AdamW |
|
|
@@ -330,16 +340,34 @@ args = Trainers::TrainingArguments.new(lr_scheduler_type: :constant, warmup_step
|
|
|
330
340
|
|
|
331
341
|
### Dataset
|
|
332
342
|
|
|
333
|
-
|
|
343
|
+
Rows are Hashes with Symbol keys. Load them from a file, or wrap an Array:
|
|
344
|
+
|
|
345
|
+
```ruby
|
|
346
|
+
dataset = Trainers::Dataset.from_jsonl("train.jsonl") # one JSON object per line
|
|
347
|
+
dataset = Trainers::Dataset.from_csv("train.csv") # header row gives the keys
|
|
348
|
+
dataset = Trainers::Dataset.from_csv("train.tsv", col_sep: "\t", converters: :numeric)
|
|
349
|
+
dataset = Trainers::Dataset.new([{ text: "A gripping film.", labels: 1 }])
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
CSV values are Strings unless you pass `converters: :numeric`. Other options go to Ruby's `CSV`.
|
|
334
353
|
|
|
335
354
|
```ruby
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
]
|
|
340
|
-
|
|
355
|
+
# A new Dataset with each row replaced. A block that doesn't return Hashes
|
|
356
|
+
# gives an Array, as Enumerable#map does.
|
|
357
|
+
dataset = dataset.map { |row| row.merge(labels: Integer(row[:label])) }
|
|
358
|
+
dataset = dataset.filter { |row| row[:text].size > 10 } # select returns an Array
|
|
359
|
+
|
|
360
|
+
# Shuffled with Random.new(seed): the same seed gives the same split
|
|
361
|
+
train, test = dataset.split(test_size: 0.2, seed: 42) # or a row count: test_size: 500
|
|
362
|
+
|
|
363
|
+
# input_ids and attention_mask, truncated to max_length tokens. The text
|
|
364
|
+
# column is dropped unless keep: true.
|
|
365
|
+
dataset = dataset.tokenize(tokenizer, column: :text, max_length: 256)
|
|
366
|
+
dataset = dataset.tokenize(tokenizer, column: [:premise, :hypothesis]) # text pairs
|
|
341
367
|
```
|
|
342
368
|
|
|
369
|
+
`tokenize` takes a transformers-rb tokenizer or a `Tokenizers::Tokenizer`.
|
|
370
|
+
|
|
343
371
|
### Data Collators
|
|
344
372
|
|
|
345
373
|
Dynamic padding collator (pads each batch to the longest sequence in that batch):
|
|
@@ -354,32 +382,61 @@ Default collator (no padding, expects uniform-length inputs):
|
|
|
354
382
|
collator = Trainers::DefaultDataCollator.new
|
|
355
383
|
```
|
|
356
384
|
|
|
357
|
-
##
|
|
385
|
+
## Support
|
|
386
|
+
|
|
387
|
+
### Models and tasks
|
|
358
388
|
|
|
359
|
-
trainers-rb
|
|
389
|
+
trainers-rb trains any `Torch::NN::Module` whose `forward` takes the batch's keys. `Trainers.from_pretrained` loads transformers-rb models through its `AutoModelFor*` classes, which cover these architectures:
|
|
360
390
|
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
|
365
|
-
|
|
|
391
|
+
<!-- architectures:start -->
|
|
392
|
+
| `model_type` | Sequence classification | Token classification | Question answering | Default LoRA targets |
|
|
393
|
+
|---|---|---|---|---|
|
|
394
|
+
| `bert` | ✓ | ✓ | – | `query`, `value` |
|
|
395
|
+
| `deberta-v2` | ✓ | – | – | `query_proj`, `value_proj` |
|
|
396
|
+
| `distilbert` | ✓ | – | ✗ no loss | `q_lin`, `v_lin` |
|
|
397
|
+
| `mpnet` | – | – | – | `q`, `v` |
|
|
398
|
+
| `xlm-roberta` | ✓ | – | – | `query`, `value` |
|
|
399
|
+
<!-- architectures:end -->
|
|
366
400
|
|
|
367
|
-
|
|
401
|
+
This table is checked against transformers-rb by the test suite. **Question answering can't be trained yet:** transformers-rb 0.2.0's `DistilBertForQuestionAnswering` raises `Todo` when given `start_positions`, so there is no loss. transformers-rb also has heads that its `AutoModelFor*` classes don't reach, such as `MPNetForSequenceClassification`. Build those directly with their config and pass them to `Trainer`:
|
|
368
402
|
|
|
369
403
|
```ruby
|
|
370
404
|
trainer = Trainers::Trainer.new(model: my_custom_model, args: args, ...)
|
|
371
405
|
```
|
|
372
406
|
|
|
373
|
-
|
|
407
|
+
### Platforms
|
|
408
|
+
|
|
409
|
+
| | Tested in CI | Hand-tested | Not supported |
|
|
410
|
+
|---|---|---|---|
|
|
411
|
+
| Ruby | 3.3, 3.4, 4.0 | | 3.2 and older |
|
|
412
|
+
| torch-rb | 0.17.1, 0.27 | | older than 0.17.1 |
|
|
413
|
+
| OS | Ubuntu 24.04, macOS 15 (Apple Silicon) | macOS 27 | Windows (untested) |
|
|
414
|
+
| CPU | ✓ | ✓ | |
|
|
415
|
+
| MPS (Apple Silicon GPU) | | DistilBERT-base with LoRA, Apple M3 Pro | |
|
|
416
|
+
| CUDA (NVIDIA GPU) | | | Untested: chosen automatically, but nobody has run it yet |
|
|
417
|
+
| Multiple GPUs | | | Not planned |
|
|
418
|
+
|
|
419
|
+
Reports for other hardware are welcome as issues, with the "Training on" log line.
|
|
420
|
+
|
|
421
|
+
### Devices
|
|
374
422
|
|
|
375
|
-
trainers-rb
|
|
423
|
+
trainers-rb picks the first available device, in this order:
|
|
424
|
+
|
|
425
|
+
- **CUDA**: NVIDIA GPU
|
|
426
|
+
- **MPS**: Apple Silicon GPU, skipped with `no_mps: true`
|
|
427
|
+
- **CPU**: always available
|
|
428
|
+
|
|
429
|
+
The first log line of a run shows what it runs on. For DistilBERT with LoRA on an Apple M3 Pro:
|
|
430
|
+
|
|
431
|
+
```
|
|
432
|
+
Training on mps (float32): 9,596 examples, 1 epoch, 600 optimizer steps (batch 16 x accumulation 1), 739,586 of 67,102,466 parameters trainable (1.10%)
|
|
433
|
+
```
|
|
376
434
|
|
|
377
|
-
-
|
|
378
|
-
- **MPS** — Apple Silicon GPU, used automatically when available
|
|
435
|
+
That run, on 10,662 movie-review sentences ([rotten_tomatoes](https://www.cs.cornell.edu/people/pabo/movie-review-data/)) with 1,066 held out, reached 82.8% accuracy in 74 seconds. A full fine-tune reached 83.6% in 121 seconds.
|
|
379
436
|
|
|
380
437
|
```ruby
|
|
381
438
|
# Force CPU
|
|
382
|
-
args = Trainers::TrainingArguments.new(
|
|
439
|
+
args = Trainers::TrainingArguments.new(device: Torch.device("cpu"))
|
|
383
440
|
|
|
384
441
|
# Or set explicitly
|
|
385
442
|
args = Trainers::TrainingArguments.new(device: Torch.device("mps"))
|
|
@@ -399,7 +456,7 @@ trainers-rb adds the training layer that transformers-rb intentionally omits. Bo
|
|
|
399
456
|
|
|
400
457
|
## Roadmap
|
|
401
458
|
|
|
402
|
-
What comes next is chosen from real fine-tunes. If something was missing in yours, [open an issue](https://github.com/Selectus2/trainers-rb/issues): a report from a user outranks this list.
|
|
459
|
+
What comes next is chosen from real fine-tunes. If something was missing in yours, [open an issue](https://github.com/Selectus2/trainers-rb/issues): a report from a user outranks this list. Planned work is tracked in the [milestones](https://github.com/Selectus2/trainers-rb/milestones).
|
|
403
460
|
|
|
404
461
|
**Not planned, at any version:** custom kernels, quantisation (including QLoRA), multi-GPU and distributed training, RL methods (GRPO, DPO), vision or speech models, GGUF export. torch-rb can't reach most of them from Ruby.
|
|
405
462
|
|
|
@@ -42,38 +42,22 @@ module Trainers
|
|
|
42
42
|
|
|
43
43
|
private
|
|
44
44
|
|
|
45
|
+
# Pads, or truncates to max_length, each sequence of Integers
|
|
45
46
|
def pad_and_stack(key, sequences)
|
|
46
|
-
max_len
|
|
47
|
-
|
|
48
|
-
pad_value = if key.to_s.include?("input_id")
|
|
49
|
-
pad_token_id
|
|
50
|
-
elsif key.to_s.include?("attention_mask")
|
|
51
|
-
0
|
|
52
|
-
elsif key.to_s.include?("label")
|
|
53
|
-
-100
|
|
54
|
-
else
|
|
55
|
-
0
|
|
56
|
-
end
|
|
47
|
+
max_len = compute_max_length(sequences.map(&:length))
|
|
48
|
+
pad_value = pad_value_for(key)
|
|
57
49
|
|
|
58
50
|
padded = sequences.map do |seq|
|
|
59
|
-
|
|
60
|
-
|
|
51
|
+
seq = seq.first(max_len)
|
|
52
|
+
seq + [pad_value] * (max_len - seq.length)
|
|
61
53
|
end
|
|
62
54
|
|
|
63
|
-
|
|
64
|
-
Torch.tensor(padded, dtype: dtype)
|
|
55
|
+
Torch.tensor(padded, dtype: :int64)
|
|
65
56
|
end
|
|
66
57
|
|
|
67
58
|
def pad_and_stack_tensors(key, tensors)
|
|
68
|
-
max_len
|
|
69
|
-
|
|
70
|
-
pad_value = if key.to_s.include?("input_id")
|
|
71
|
-
pad_token_id
|
|
72
|
-
elsif key.to_s.include?("attention_mask")
|
|
73
|
-
0
|
|
74
|
-
else
|
|
75
|
-
0
|
|
76
|
-
end
|
|
59
|
+
max_len = compute_max_length(tensors.map { |t| t.size(0) })
|
|
60
|
+
pad_value = pad_value_for(key)
|
|
77
61
|
|
|
78
62
|
padded = tensors.map do |t|
|
|
79
63
|
if t.size(0) < max_len
|
|
@@ -87,6 +71,18 @@ module Trainers
|
|
|
87
71
|
Torch.stack(padded)
|
|
88
72
|
end
|
|
89
73
|
|
|
74
|
+
# Labels pad with -100, which cross-entropy ignores
|
|
75
|
+
def pad_value_for(key)
|
|
76
|
+
name = key.to_s
|
|
77
|
+
if name.include?("input_id")
|
|
78
|
+
pad_token_id
|
|
79
|
+
elsif name.include?("label")
|
|
80
|
+
-100
|
|
81
|
+
else
|
|
82
|
+
0
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
|
|
90
86
|
def compute_max_length(lengths)
|
|
91
87
|
max_len = @max_length || lengths.max
|
|
92
88
|
if @pad_to_multiple_of
|
|
@@ -1,11 +1,47 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "json"
|
|
4
|
+
|
|
3
5
|
module Trainers
|
|
6
|
+
# Rows for training and evaluation: an Array of Hashes with Symbol keys,
|
|
7
|
+
# the shape the collators and Trainer expect.
|
|
8
|
+
#
|
|
9
|
+
# ds = Trainers::Dataset.from_jsonl("train.jsonl")
|
|
10
|
+
# train, eval = ds.split(test_size: 0.1)
|
|
11
|
+
# train = train.map { |row| row.merge(labels: Integer(row[:label])) }
|
|
4
12
|
class Dataset
|
|
5
13
|
include Enumerable
|
|
6
14
|
|
|
7
15
|
attr_reader :data
|
|
8
16
|
|
|
17
|
+
# One JSON object per line. Blank lines are skipped.
|
|
18
|
+
def self.from_jsonl(path)
|
|
19
|
+
rows = []
|
|
20
|
+
File.foreach(path).with_index(1) do |line, number|
|
|
21
|
+
next if line.strip.empty?
|
|
22
|
+
|
|
23
|
+
row = begin
|
|
24
|
+
JSON.parse(line, symbolize_names: true)
|
|
25
|
+
rescue JSON::ParserError => e
|
|
26
|
+
raise ConfigurationError, "#{path}:#{number} is not valid JSON: #{e.message}"
|
|
27
|
+
end
|
|
28
|
+
raise ConfigurationError, "#{path}:#{number} is a JSON #{row.class}, not an object" unless row.is_a?(Hash)
|
|
29
|
+
|
|
30
|
+
rows << row
|
|
31
|
+
end
|
|
32
|
+
new(rows)
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# A CSV file whose header row gives the keys. Values are Strings unless
|
|
36
|
+
# csv_options say otherwise, e.g. converters: :numeric. Other options
|
|
37
|
+
# (col_sep:, encoding: ...) go to CSV.
|
|
38
|
+
def self.from_csv(path, **csv_options)
|
|
39
|
+
require "csv"
|
|
40
|
+
|
|
41
|
+
table = CSV.read(path, headers: true, header_converters: :symbol, **csv_options)
|
|
42
|
+
new(table.map(&:to_h))
|
|
43
|
+
end
|
|
44
|
+
|
|
9
45
|
def initialize(data)
|
|
10
46
|
@data = data
|
|
11
47
|
end
|
|
@@ -22,5 +58,118 @@ module Trainers
|
|
|
22
58
|
def each(&block)
|
|
23
59
|
@data.each(&block)
|
|
24
60
|
end
|
|
61
|
+
|
|
62
|
+
# A new Dataset when the block returns a Hash for every row. Any other
|
|
63
|
+
# result gives an Array, as Enumerable#map does. The block gets a copy of
|
|
64
|
+
# each row, so row.delete(:label) leaves this Dataset unchanged.
|
|
65
|
+
def map(&block)
|
|
66
|
+
return enum_for(:map) unless block
|
|
67
|
+
|
|
68
|
+
results = @data.map { |row| block.call(row.is_a?(Hash) ? row.dup : row) }
|
|
69
|
+
results.all?(Hash) ? self.class.new(results) : results
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# A new Dataset with the rows the block returns true for. select still
|
|
73
|
+
# returns an Array, as Enumerable#select does.
|
|
74
|
+
def filter(&block)
|
|
75
|
+
return enum_for(:filter) unless block
|
|
76
|
+
|
|
77
|
+
self.class.new(@data.select(&block))
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
TOKENIZE_BATCH = 1000
|
|
81
|
+
|
|
82
|
+
# A new Dataset with input_ids and attention_mask (and token_type_ids
|
|
83
|
+
# when the tokenizer gives them) for the text in column, truncated to
|
|
84
|
+
# max_length tokens. Pass two columns for text pairs, e.g. NLI. The text
|
|
85
|
+
# columns are dropped unless keep: true. Takes a Tokenizers::Tokenizer or
|
|
86
|
+
# a transformers-rb tokenizer.
|
|
87
|
+
def tokenize(tokenizer, column: :text, max_length: 512, keep: false)
|
|
88
|
+
columns = Array(column).map(&:to_sym)
|
|
89
|
+
unless columns.size.between?(1, 2)
|
|
90
|
+
raise ArgumentError, "column must name one text column, or two for text pairs, got #{column.inspect}"
|
|
91
|
+
end
|
|
92
|
+
unless max_length.is_a?(Integer) && max_length.positive?
|
|
93
|
+
raise ArgumentError, "max_length must be a positive Integer, got #{max_length.inspect}"
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
texts = @data.each_with_index.map do |row, index|
|
|
97
|
+
columns.map do |name|
|
|
98
|
+
text = row[name]
|
|
99
|
+
raise ConfigurationError, "Row #{index} has no String in #{name.inspect}: #{text.inspect}" unless text.is_a?(String)
|
|
100
|
+
|
|
101
|
+
text
|
|
102
|
+
end
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
rows = []
|
|
106
|
+
texts.each_slice(TOKENIZE_BATCH).with_index do |slice, n|
|
|
107
|
+
encoded = encode_texts(tokenizer, slice, max_length)
|
|
108
|
+
slice.each_index do |i|
|
|
109
|
+
row = @data[n * TOKENIZE_BATCH + i]
|
|
110
|
+
row = row.except(*columns) unless keep
|
|
111
|
+
rows << row.merge(encoded[i])
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
self.class.new(rows)
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
# Shuffles with Random.new(seed) and returns [train, test]. test_size is
|
|
118
|
+
# a fraction in (0, 1), or a row count.
|
|
119
|
+
def split(test_size:, seed: 42)
|
|
120
|
+
count = case test_size
|
|
121
|
+
when Float
|
|
122
|
+
raise ArgumentError, "test_size must be between 0 and 1, got #{test_size}" unless test_size > 0 && test_size < 1
|
|
123
|
+
|
|
124
|
+
(size * test_size).round
|
|
125
|
+
when Integer
|
|
126
|
+
test_size
|
|
127
|
+
else
|
|
128
|
+
raise ArgumentError, "test_size must be a Float in (0, 1) or an Integer row count, got #{test_size.inspect}"
|
|
129
|
+
end
|
|
130
|
+
unless count.between?(1, size - 1)
|
|
131
|
+
raise ArgumentError, "test_size #{test_size.inspect} leaves #{count} of #{size} rows for testing; " \
|
|
132
|
+
"both parts need at least one row"
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
rows = @data.shuffle(random: Random.new(seed))
|
|
136
|
+
[self.class.new(rows.drop(count)), self.class.new(rows.first(count))]
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
private
|
|
140
|
+
|
|
141
|
+
# One Hash of model inputs per text (or [text, pair])
|
|
142
|
+
def encode_texts(tokenizer, texts, max_length)
|
|
143
|
+
if tokenizer.respond_to?(:backend_tokenizer)
|
|
144
|
+
# transformers-rb tokenizer
|
|
145
|
+
first, pairs = texts.transpose
|
|
146
|
+
encoded = tokenizer.(first, text_pair: pairs, truncation: true, max_length: max_length)
|
|
147
|
+
keys = %i[input_ids attention_mask token_type_ids].select { |key| encoded[key] }
|
|
148
|
+
texts.each_index.map { |i| keys.to_h { |key| [key, encoded[key][i]] } }
|
|
149
|
+
elsif tokenizer.respond_to?(:encode_batch)
|
|
150
|
+
encode_with_backend(tokenizer, texts, max_length)
|
|
151
|
+
else
|
|
152
|
+
raise ConfigurationError, "#{tokenizer.class} is not a Tokenizers::Tokenizer or a transformers-rb tokenizer"
|
|
153
|
+
end
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
# Tokenizers::Tokenizer keeps truncation as a setting, so set it for the
|
|
157
|
+
# call and put the caller's back afterwards
|
|
158
|
+
def encode_with_backend(tokenizer, texts, max_length)
|
|
159
|
+
previous = tokenizer.truncation
|
|
160
|
+
tokenizer.enable_truncation(max_length)
|
|
161
|
+
inputs = texts.map { |text| text.size == 1 ? text.first : text }
|
|
162
|
+
tokenizer.encode_batch(inputs).map do |encoding|
|
|
163
|
+
row = { input_ids: encoding.ids, attention_mask: encoding.attention_mask }
|
|
164
|
+
row[:token_type_ids] = encoding.type_ids if texts.first.size == 2
|
|
165
|
+
row
|
|
166
|
+
end
|
|
167
|
+
ensure
|
|
168
|
+
tokenizer.no_truncation
|
|
169
|
+
if previous
|
|
170
|
+
options = previous.reject { |key, _| key == "max_length" }.transform_keys(&:to_sym)
|
|
171
|
+
tokenizer.enable_truncation(previous["max_length"], **options)
|
|
172
|
+
end
|
|
173
|
+
end
|
|
25
174
|
end
|
|
26
175
|
end
|
|
@@ -9,12 +9,25 @@ module Trainers
|
|
|
9
9
|
r: 8,
|
|
10
10
|
lora_alpha: 16,
|
|
11
11
|
lora_dropout: 0.0,
|
|
12
|
-
target_modules:
|
|
12
|
+
target_modules: nil, # nil: DEFAULT_TARGETS for the model; or names, or :all_linear
|
|
13
13
|
bias: :none, # :none, :all, :lora_only
|
|
14
14
|
task_type: :sequence_classification,
|
|
15
15
|
modules_to_save: nil # nil: the task head for task_type
|
|
16
16
|
}.freeze
|
|
17
17
|
|
|
18
|
+
# Attention query and value projections for each transformers-rb
|
|
19
|
+
# model_type, used when target_modules is nil
|
|
20
|
+
DEFAULT_TARGETS = {
|
|
21
|
+
"bert" => %w[query value],
|
|
22
|
+
"xlm-roberta" => %w[query value],
|
|
23
|
+
"distilbert" => %w[q_lin v_lin],
|
|
24
|
+
"deberta-v2" => %w[query_proj value_proj],
|
|
25
|
+
"mpnet" => %w[q v]
|
|
26
|
+
}.freeze
|
|
27
|
+
|
|
28
|
+
# Targets for models without a transformers config, the default before 0.4.0
|
|
29
|
+
FALLBACK_TARGETS = %w[query value].freeze
|
|
30
|
+
|
|
18
31
|
# Modules that stay fully trainable for each task_type, so the task head
|
|
19
32
|
# (randomly initialised by from_pretrained) learns along with the adapters
|
|
20
33
|
TASK_HEADS = {
|
|
@@ -36,6 +49,24 @@ module Trainers
|
|
|
36
49
|
Array(@modules_to_save.nil? ? TASK_HEADS.fetch(@task_type&.to_sym, []) : @modules_to_save).map(&:to_s)
|
|
37
50
|
end
|
|
38
51
|
|
|
52
|
+
# target_modules when given, otherwise the defaults for the model's
|
|
53
|
+
# config.model_type, or FALLBACK_TARGETS for a model without a config.
|
|
54
|
+
# Returns [targets, source], source being :given, :model_type or :fallback.
|
|
55
|
+
def resolved_target_modules(model)
|
|
56
|
+
return [@target_modules, :given] unless @target_modules.nil?
|
|
57
|
+
|
|
58
|
+
config = model.respond_to?(:config) ? model.config : nil
|
|
59
|
+
return [FALLBACK_TARGETS, :fallback] unless config.respond_to?(:model_type)
|
|
60
|
+
|
|
61
|
+
type = config.model_type.to_s
|
|
62
|
+
targets = DEFAULT_TARGETS[type]
|
|
63
|
+
unless targets
|
|
64
|
+
raise AdapterError, "No default target_modules for model_type #{type.inspect}. Pass target_modules: with " \
|
|
65
|
+
"the Linear modules to adapt, from: #{LoraUtils.linear_names(model).first(20).join(', ')}"
|
|
66
|
+
end
|
|
67
|
+
[targets, :model_type]
|
|
68
|
+
end
|
|
69
|
+
|
|
39
70
|
def scaling
|
|
40
71
|
@lora_alpha.to_f / @r
|
|
41
72
|
end
|
|
@@ -18,11 +18,13 @@ module Trainers
|
|
|
18
18
|
model.parameters.each { |p| p.requires_grad = false }
|
|
19
19
|
|
|
20
20
|
# Find and replace target modules
|
|
21
|
-
|
|
21
|
+
# The per-architecture defaults are exact names, so they never fall back to substrings
|
|
22
|
+
target_modules, source = config.resolved_target_modules(model)
|
|
23
|
+
targets = LoraUtils.find_target_modules(model, target_modules, fallback: source != :model_type)
|
|
22
24
|
|
|
23
25
|
if targets.empty?
|
|
24
|
-
raise AdapterError, "No Linear modules found matching target_modules: #{
|
|
25
|
-
"
|
|
26
|
+
raise AdapterError, "No Linear modules found matching target_modules: #{target_modules.inspect}. " \
|
|
27
|
+
"Linear modules: #{LoraUtils.linear_names(model).first(20).join(', ')}"
|
|
26
28
|
end
|
|
27
29
|
|
|
28
30
|
targets.each do |name, linear|
|
|
@@ -53,7 +55,8 @@ module Trainers
|
|
|
53
55
|
|
|
54
56
|
unfreeze_modules_to_save(model, config)
|
|
55
57
|
|
|
56
|
-
Trainers.log(:info, "LoRA
|
|
58
|
+
Trainers.log(:info, "LoRA target_modules #{target_modules.inspect}#{' (default)' unless source == :given} " \
|
|
59
|
+
"applied to #{targets.size} modules: #{targets.keys.join(', ')}")
|
|
57
60
|
LoraUtils.print_trainable_parameters(model)
|
|
58
61
|
|
|
59
62
|
model
|
|
@@ -2,23 +2,36 @@
|
|
|
2
2
|
|
|
3
3
|
module Trainers
|
|
4
4
|
module LoraUtils
|
|
5
|
-
# Find
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
5
|
+
# Find the Linear modules matching target_modules, as PEFT does: a name
|
|
6
|
+
# matches a target when it is the target or ends with ".target". A target
|
|
7
|
+
# that matches nothing that way falls back to matching any name that
|
|
8
|
+
# contains it, with a deprecation warning; substring matching ends in 0.5.0.
|
|
9
|
+
# fallback: false turns the fallback off (for the per-architecture defaults).
|
|
10
|
+
def self.find_target_modules(model, target_modules, fallback: true)
|
|
11
|
+
linears = model.named_modules.select { |_, mod| mod.is_a?(Torch::NN::Linear) }
|
|
12
|
+
return linears.to_h if target_modules == :all_linear
|
|
11
13
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
14
|
+
targets = {}
|
|
15
|
+
Array(target_modules).map(&:to_s).each do |pattern|
|
|
16
|
+
matched = linears.select { |name, _| name == pattern || name.end_with?(".#{pattern}") }
|
|
17
|
+
if matched.empty? && fallback
|
|
18
|
+
matched = linears.select { |name, _| name.include?(pattern) }
|
|
19
|
+
unless matched.empty?
|
|
20
|
+
Trainers.log(:warn, "target_modules #{pattern.inspect} matched #{matched.map(&:first).join(', ')} by substring. " \
|
|
21
|
+
"Substring matching is deprecated and ends in 0.5.0; name the modules by their last part " \
|
|
22
|
+
"or full name instead.")
|
|
17
23
|
end
|
|
18
24
|
end
|
|
25
|
+
matched.each { |name, mod| targets[name] = mod }
|
|
19
26
|
end
|
|
20
27
|
|
|
21
|
-
|
|
28
|
+
# Keep the model's module order
|
|
29
|
+
linears.select { |name, _| targets.key?(name) }.to_h
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
# Names of the Linear modules, for error messages
|
|
33
|
+
def self.linear_names(model)
|
|
34
|
+
model.named_modules.select { |_, mod| mod.is_a?(Torch::NN::Linear) }.map(&:first)
|
|
22
35
|
end
|
|
23
36
|
|
|
24
37
|
# Replace a child module on a parent by setting the instance variable.
|
data/lib/trainers/trainer.rb
CHANGED
|
@@ -46,20 +46,31 @@ module Trainers
|
|
|
46
46
|
# One step is one optimizer update. The last accumulation window of an
|
|
47
47
|
# epoch may be shorter than gradient_accumulation_steps; it still steps.
|
|
48
48
|
steps_per_epoch = (batches_per_epoch.to_f / accum).ceil
|
|
49
|
-
|
|
49
|
+
if @args.max_steps
|
|
50
|
+
total_steps = @args.max_steps
|
|
51
|
+
num_epochs = (total_steps.to_f / steps_per_epoch).ceil
|
|
52
|
+
else
|
|
53
|
+
num_epochs = @args.num_train_epochs
|
|
54
|
+
total_steps = steps_per_epoch * num_epochs
|
|
55
|
+
end
|
|
50
56
|
|
|
51
57
|
@state.max_steps = total_steps
|
|
52
|
-
@state.num_train_epochs =
|
|
58
|
+
@state.num_train_epochs = num_epochs
|
|
53
59
|
|
|
54
60
|
@optimizer = create_optimizer
|
|
55
61
|
@lr_scheduler = create_scheduler(total_steps)
|
|
56
62
|
@optimizer.zero_grad
|
|
63
|
+
log_training_setup(device, num_epochs, total_steps)
|
|
57
64
|
start_epoch, skip_batches = resume_from_checkpoint
|
|
58
65
|
|
|
59
66
|
@callback_handler.fire(:on_train_begin, @args, @state, @control)
|
|
60
67
|
|
|
61
|
-
|
|
68
|
+
# With max_steps, epochs continue until the step budget is spent, even
|
|
69
|
+
# when callbacks end some epochs early with should_epoch_stop
|
|
70
|
+
(0..).each do |epoch|
|
|
71
|
+
break if @args.max_steps.nil? && epoch >= num_epochs
|
|
62
72
|
next if epoch < start_epoch
|
|
73
|
+
break if @state.global_step >= total_steps # done, or resumed from the final checkpoint
|
|
63
74
|
|
|
64
75
|
@state.epoch = epoch + 1
|
|
65
76
|
@callback_handler.fire(:on_epoch_begin, @args, @state, @control)
|
|
@@ -87,6 +98,7 @@ module Trainers
|
|
|
87
98
|
|
|
88
99
|
epoch_loss += loss.item
|
|
89
100
|
epoch_batches += 1
|
|
101
|
+
release_memory
|
|
90
102
|
|
|
91
103
|
next unless position == window_size - 1
|
|
92
104
|
|
|
@@ -95,6 +107,7 @@ module Trainers
|
|
|
95
107
|
@lr_scheduler.step
|
|
96
108
|
@optimizer.zero_grad
|
|
97
109
|
@state.global_step += 1
|
|
110
|
+
@control.should_training_stop = true if @state.global_step >= total_steps
|
|
98
111
|
|
|
99
112
|
# Logging
|
|
100
113
|
if should_log?
|
|
@@ -167,42 +180,39 @@ module Trainers
|
|
|
167
180
|
raise ArgumentError, "No eval_dataset provided" unless dataset
|
|
168
181
|
|
|
169
182
|
device = @args.resolved_device
|
|
183
|
+
@model.to(device)
|
|
170
184
|
@model.eval
|
|
171
185
|
|
|
172
186
|
all_preds = []
|
|
173
187
|
all_labels = []
|
|
174
188
|
total_loss = 0.0
|
|
175
|
-
|
|
189
|
+
examples = 0
|
|
176
190
|
|
|
177
191
|
Torch.no_grad do
|
|
178
192
|
each_batch(dataset, @args.per_device_eval_batch_size) do |batch|
|
|
179
|
-
batch
|
|
180
|
-
labels =
|
|
181
|
-
|
|
182
|
-
output = forward(batch)
|
|
193
|
+
batch = move_to_device(batch, device)
|
|
194
|
+
output, labels = forward_with_labels(batch)
|
|
195
|
+
logits = logits_of(output)
|
|
183
196
|
|
|
184
197
|
if labels
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
total_loss +=
|
|
198
|
+
# Weight by batch size, so a short last batch counts for its rows only
|
|
199
|
+
size = labels.size(0)
|
|
200
|
+
total_loss += loss_for(output, logits, labels).item * size
|
|
201
|
+
examples += size
|
|
188
202
|
all_labels << labels.detach.cpu
|
|
189
203
|
end
|
|
190
|
-
total_steps += 1
|
|
191
|
-
|
|
192
|
-
logits = output.respond_to?(:logits) ? output.logits : output
|
|
193
204
|
all_preds << logits.detach.cpu
|
|
205
|
+
release_memory
|
|
194
206
|
end
|
|
195
207
|
end
|
|
196
208
|
|
|
197
209
|
@model.train
|
|
198
210
|
|
|
199
211
|
metrics = {}
|
|
200
|
-
metrics[:eval_loss] = total_loss /
|
|
212
|
+
metrics[:eval_loss] = total_loss / examples if examples > 0
|
|
201
213
|
|
|
202
214
|
if @compute_metrics && all_preds.any? && all_labels.any?
|
|
203
|
-
|
|
204
|
-
labels = Torch.cat(all_labels)
|
|
205
|
-
eval_pred = EvalPrediction.new(predictions: preds, label_ids: labels)
|
|
215
|
+
eval_pred = EvalPrediction.new(predictions: concat_padded(all_preds), label_ids: concat_padded(all_labels))
|
|
206
216
|
custom_metrics = @compute_metrics.call(eval_pred)
|
|
207
217
|
metrics.merge!(custom_metrics)
|
|
208
218
|
end
|
|
@@ -210,21 +220,25 @@ module Trainers
|
|
|
210
220
|
metrics
|
|
211
221
|
end
|
|
212
222
|
|
|
223
|
+
# Logits for every row. Batches padded to different lengths, as in token
|
|
224
|
+
# classification, are padded to the longest with -100.
|
|
213
225
|
def predict(test_dataset)
|
|
214
226
|
device = @args.resolved_device
|
|
227
|
+
@model.to(device)
|
|
215
228
|
@model.eval
|
|
216
229
|
|
|
217
230
|
all_preds = []
|
|
218
231
|
Torch.no_grad do
|
|
219
232
|
each_batch(test_dataset, @args.per_device_eval_batch_size) do |batch|
|
|
220
|
-
batch
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
233
|
+
batch = move_to_device(batch, device)
|
|
234
|
+
pop_labels(batch)
|
|
235
|
+
all_preds << logits_of(forward(batch)).detach.cpu
|
|
236
|
+
release_memory
|
|
224
237
|
end
|
|
225
238
|
end
|
|
239
|
+
@model.train
|
|
226
240
|
|
|
227
|
-
|
|
241
|
+
concat_padded(all_preds)
|
|
228
242
|
end
|
|
229
243
|
|
|
230
244
|
def save_model(output_dir = nil)
|
|
@@ -234,35 +248,103 @@ module Trainers
|
|
|
234
248
|
|
|
235
249
|
private
|
|
236
250
|
|
|
251
|
+
# One line with what is about to run, so a run on the wrong device or
|
|
252
|
+
# with the wrong number of trainable weights shows up before it starts
|
|
253
|
+
def log_training_setup(device, num_epochs, total_steps)
|
|
254
|
+
params = @model.parameters
|
|
255
|
+
total = params.sum(&:numel)
|
|
256
|
+
trainable = params.select(&:requires_grad).sum(&:numel)
|
|
257
|
+
dtype = params.first&.dtype || "no parameters"
|
|
258
|
+
percent = total.positive? ? trainable * 100.0 / total : 0.0
|
|
259
|
+
|
|
260
|
+
Trainers.log(:info, "Training on #{device.respond_to?(:type) ? device.type : device} (#{dtype}): #{LoraUtils.format_number(@train_dataset.size)} examples, " \
|
|
261
|
+
"#{num_epochs} epoch#{'s' unless num_epochs == 1}, #{LoraUtils.format_number(total_steps)} optimizer steps " \
|
|
262
|
+
"(batch #{@args.per_device_train_batch_size} x accumulation #{@args.gradient_accumulation_steps}), " \
|
|
263
|
+
"#{LoraUtils.format_number(trainable)} of #{LoraUtils.format_number(total)} " \
|
|
264
|
+
"parameters trainable (#{format('%.2f', percent)}%)")
|
|
265
|
+
end
|
|
266
|
+
|
|
237
267
|
def compute_loss(batch)
|
|
268
|
+
output, labels = forward_with_labels(batch)
|
|
269
|
+
if labels.nil? && !(output.respond_to?(:loss) && output.loss)
|
|
270
|
+
raise ConfigurationError, "Model did not return a loss and no labels found in batch. " \
|
|
271
|
+
"Either pass labels in your dataset or use a model that computes loss."
|
|
272
|
+
end
|
|
273
|
+
|
|
274
|
+
loss_for(output, logits_of(output), labels)
|
|
275
|
+
end
|
|
276
|
+
|
|
277
|
+
# Runs the model with the labels, so models that compute their own loss
|
|
278
|
+
# return it. Returns [output, labels]; labels is nil when the batch has none.
|
|
279
|
+
def forward_with_labels(batch)
|
|
238
280
|
label_values = pop_labels(batch)
|
|
239
281
|
labels = label_values.values.first
|
|
240
282
|
|
|
241
|
-
# Try passing labels to the model (some models compute loss internally)
|
|
242
283
|
output = begin
|
|
243
284
|
forward(batch.merge(label_values))
|
|
285
|
+
rescue NameError => e
|
|
286
|
+
# transformers-rb 0.2.0's BertForTokenClassification computes its loss
|
|
287
|
+
# with an unqualified CrossEntropyLoss and raises whenever it gets labels
|
|
288
|
+
raise unless e.name == :CrossEntropyLoss
|
|
289
|
+
|
|
290
|
+
forward(batch)
|
|
291
|
+
rescue ArgumentError => e
|
|
292
|
+
# A model whose forward takes no label keyword: run it without labels
|
|
293
|
+
# and compute the loss outside it
|
|
294
|
+
raise unless e.message.match?(/unknown keywords?:/) && label_values.keys.any? { |key| e.message.include?(":#{key}") }
|
|
295
|
+
|
|
296
|
+
forward(batch)
|
|
244
297
|
rescue => e
|
|
245
298
|
# If the model doesn't support labels kwarg (e.g. transformers-rb Todo),
|
|
246
299
|
# fall back to forward without labels + external loss
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
raise
|
|
251
|
-
end
|
|
300
|
+
raise unless e.message.include?("Todo") || e.message.include?("not implemented")
|
|
301
|
+
|
|
302
|
+
forward(batch)
|
|
252
303
|
end
|
|
253
304
|
|
|
254
|
-
|
|
255
|
-
|
|
305
|
+
[output, labels]
|
|
306
|
+
end
|
|
256
307
|
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
308
|
+
# The model's own loss when it returns one, otherwise cross-entropy over
|
|
309
|
+
# the logits. Per-token logits [batch, seq, classes] are flattened, and
|
|
310
|
+
# labels of -100 (padding) are ignored.
|
|
311
|
+
def loss_for(output, logits, labels)
|
|
312
|
+
return output.loss if output.respond_to?(:loss) && output.loss
|
|
313
|
+
|
|
314
|
+
if logits.dim > 2
|
|
315
|
+
Torch::NN::F.cross_entropy(logits.reshape(-1, logits.size(-1)), labels.reshape(-1))
|
|
262
316
|
else
|
|
263
|
-
|
|
264
|
-
|
|
317
|
+
Torch::NN::F.cross_entropy(logits, labels)
|
|
318
|
+
end
|
|
319
|
+
end
|
|
320
|
+
|
|
321
|
+
# torch-rb tensors are freed when Ruby garbage-collects them, and Ruby
|
|
322
|
+
# cannot see the memory LibTorch holds for them, so it rarely collects.
|
|
323
|
+
# Each batch's activations would then pile up: evaluating DistilBERT on
|
|
324
|
+
# MPS reached 22 GiB in 17 batches. A minor collection per batch keeps
|
|
325
|
+
# memory flat and costs under a millisecond.
|
|
326
|
+
def release_memory
|
|
327
|
+
GC.start(full_mark: false, immediate_sweep: true)
|
|
328
|
+
end
|
|
329
|
+
|
|
330
|
+
def logits_of(output)
|
|
331
|
+
output.respond_to?(:logits) ? output.logits : output
|
|
332
|
+
end
|
|
333
|
+
|
|
334
|
+
# Concatenates per-batch tensors along dim 0, first padding dim 1 to the
|
|
335
|
+
# longest batch with -100
|
|
336
|
+
def concat_padded(tensors)
|
|
337
|
+
return Torch.cat(tensors) if tensors.first.dim < 2
|
|
338
|
+
|
|
339
|
+
length = tensors.map { |t| t.size(1) }.max
|
|
340
|
+
padded = tensors.map do |t|
|
|
341
|
+
next t if t.size(1) == length
|
|
342
|
+
|
|
343
|
+
shape = t.shape.dup
|
|
344
|
+
shape[1] = length - t.size(1)
|
|
345
|
+
Torch.cat([t, Torch.full(shape, -100, dtype: t.dtype)], 1)
|
|
265
346
|
end
|
|
347
|
+
Torch.cat(padded)
|
|
266
348
|
end
|
|
267
349
|
|
|
268
350
|
def forward(batch)
|
|
@@ -5,6 +5,7 @@ module Trainers
|
|
|
5
5
|
DEFAULTS = {
|
|
6
6
|
output_dir: "./output",
|
|
7
7
|
num_train_epochs: 3,
|
|
8
|
+
max_steps: nil, # optimizer steps; overrides num_train_epochs when set
|
|
8
9
|
per_device_train_batch_size: 8,
|
|
9
10
|
per_device_eval_batch_size: 8,
|
|
10
11
|
learning_rate: 5e-5,
|
|
@@ -56,10 +57,14 @@ module Trainers
|
|
|
56
57
|
validate!
|
|
57
58
|
end
|
|
58
59
|
|
|
60
|
+
# The device training runs on: device: when given, otherwise CUDA, then
|
|
61
|
+
# MPS (unless no_mps:), then the CPU
|
|
59
62
|
def resolved_device
|
|
60
63
|
return @device if @device
|
|
61
64
|
|
|
62
|
-
if
|
|
65
|
+
if Torch::CUDA.available?
|
|
66
|
+
Torch.device("cuda")
|
|
67
|
+
elsif !@no_mps && defined?(Torch::Backends::MPS) && Torch::Backends::MPS.available?
|
|
63
68
|
Torch.device("mps")
|
|
64
69
|
else
|
|
65
70
|
Torch.device("cpu")
|
|
@@ -88,6 +93,7 @@ module Trainers
|
|
|
88
93
|
end
|
|
89
94
|
check(:warmup_steps, "a non-negative Integer") { |v| v.is_a?(Integer) && !v.negative? }
|
|
90
95
|
check(:warmup_ratio, "a number between 0 and 1") { |v| v.is_a?(Numeric) && v.between?(0, 1) }
|
|
96
|
+
check(:max_steps, "nil or a positive Integer") { |v| v.nil? || (v.is_a?(Integer) && v.positive?) }
|
|
91
97
|
check(:eval_steps, "nil or a positive Integer") { |v| v.nil? || (v.is_a?(Integer) && v.positive?) }
|
|
92
98
|
check(:seed, "nil or a non-negative Integer") { |v| v.nil? || (v.is_a?(Integer) && !v.negative?) }
|
|
93
99
|
check(:resume_from_checkpoint, "nil, true or a checkpoint directory String") { |v| v.nil? || v == true || v.is_a?(String) }
|
data/lib/trainers/version.rb
CHANGED
data/lib/trainers.rb
CHANGED
|
@@ -31,10 +31,15 @@ module Trainers
|
|
|
31
31
|
seed
|
|
32
32
|
end
|
|
33
33
|
|
|
34
|
-
# Convenience method: load model + tokenizer and prepare for training
|
|
35
|
-
|
|
34
|
+
# Convenience method: load model + tokenizer and prepare for training.
|
|
35
|
+
# labels: names the classes, in id order, so the saved config.json carries
|
|
36
|
+
# id2label and label2id; it sets num_labels to labels.size.
|
|
37
|
+
def self.from_pretrained(model_name, task: :sequence_classification, num_labels: nil, labels: nil)
|
|
36
38
|
require "transformers-rb"
|
|
37
39
|
|
|
40
|
+
labels = label_names(labels, num_labels)
|
|
41
|
+
num_labels ||= labels ? labels.size : 2
|
|
42
|
+
|
|
38
43
|
model_class = case task
|
|
39
44
|
when :sequence_classification
|
|
40
45
|
Transformers::AutoModelForSequenceClassification
|
|
@@ -46,12 +51,39 @@ module Trainers
|
|
|
46
51
|
Transformers::AutoModel
|
|
47
52
|
end
|
|
48
53
|
|
|
54
|
+
if task == :question_answering
|
|
55
|
+
Trainers.log(:warn, "transformers-rb #{Transformers::VERSION} cannot compute the question-answering loss " \
|
|
56
|
+
"(DistilBertForQuestionAnswering raises Todo when given start_positions), so this model " \
|
|
57
|
+
"can predict but not be trained")
|
|
58
|
+
end
|
|
59
|
+
|
|
49
60
|
model = model_class.from_pretrained(model_name, num_labels: num_labels)
|
|
50
61
|
tokenizer = Transformers::AutoTokenizer.from_pretrained(model_name)
|
|
51
62
|
|
|
63
|
+
# transformers-rb 0.2.0 config has no id2label or label2id writer, and
|
|
64
|
+
# from_pretrained drops them when passed as options
|
|
65
|
+
if labels
|
|
66
|
+
model.config.instance_variable_set(:@id2label, labels.each_with_index.to_h { |name, id| [id, name] })
|
|
67
|
+
model.config.instance_variable_set(:@label2id, labels.each_with_index.to_h)
|
|
68
|
+
end
|
|
69
|
+
|
|
52
70
|
[model, tokenizer]
|
|
53
71
|
end
|
|
54
72
|
|
|
73
|
+
# Internal: labels as unique Strings, checked against num_labels
|
|
74
|
+
def self.label_names(labels, num_labels)
|
|
75
|
+
return if labels.nil?
|
|
76
|
+
|
|
77
|
+
names = Array(labels).map(&:to_s)
|
|
78
|
+
raise ArgumentError, "labels must be a non-empty list of unique names, got #{labels.inspect}" if names.empty? || names.uniq.size != names.size
|
|
79
|
+
if num_labels && num_labels != names.size
|
|
80
|
+
raise ArgumentError, "num_labels: #{num_labels} contradicts labels: #{names.inspect} (#{names.size} labels)"
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
names
|
|
84
|
+
end
|
|
85
|
+
private_class_method :label_names
|
|
86
|
+
|
|
55
87
|
# Loads a directory written by Trainer#save_model. Returns [model, tokenizer].
|
|
56
88
|
def self.load_pretrained(model_dir)
|
|
57
89
|
SaveUtils.load_pretrained(model_dir)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: trainers-rb
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Vishwajeetsingh Desurkar
|
|
@@ -80,6 +80,20 @@ dependencies:
|
|
|
80
80
|
- - ">="
|
|
81
81
|
- !ruby/object:Gem::Version
|
|
82
82
|
version: '1.5'
|
|
83
|
+
- !ruby/object:Gem::Dependency
|
|
84
|
+
name: csv
|
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
|
86
|
+
requirements:
|
|
87
|
+
- - ">="
|
|
88
|
+
- !ruby/object:Gem::Version
|
|
89
|
+
version: '3.2'
|
|
90
|
+
type: :runtime
|
|
91
|
+
prerelease: false
|
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
93
|
+
requirements:
|
|
94
|
+
- - ">="
|
|
95
|
+
- !ruby/object:Gem::Version
|
|
96
|
+
version: '3.2'
|
|
83
97
|
- !ruby/object:Gem::Dependency
|
|
84
98
|
name: rake
|
|
85
99
|
requirement: !ruby/object:Gem::Requirement
|