trainers-rb 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 19b0d8f13145e80f1e129e39e929910a02af3767500fe85b9fc6def193c8808e
4
- data.tar.gz: f32c79f763af6137051f95582b1692444b7dba9a863df0e665bf12b8b9f7463e
3
+ metadata.gz: 63872fbc41baf9c23b43cb48722c0f5a94becef7480c2452e5a51ba7b10ac817
4
+ data.tar.gz: 99cbdb6695ae3952cdcf37e991cb5d9e1471b49fa3d5346bc2a26bc88575354d
5
5
  SHA512:
6
- metadata.gz: 301a7d6777e2ad1751236c4f79389812f05b47ef891954184c422434d5bbfe9ab65ed8c6af462aeeaba0c58f1832935abe585d2da07d1d09bc2b06649b5b9985
7
- data.tar.gz: 63f1e7ce998253502a7fd6d78d2295212d0bc07fdf1f1a9d78a7202d2ee38c887540ecc54af394234112f7903aacf83100b0dfd0f1541e3042ed1382316edf2f
6
+ metadata.gz: '080c41aa2ba050105595f0ce4fadcc305daba493dae923bebe9e5a8610ab8c6138d48cafb34cd7445eb43b88663f1f45bf66a5c64043d70eb487d34812b34073'
7
+ data.tar.gz: 1d94c9069209dfa6a64d418e9d26fb7baa9329ac83514ab54848731dc6cd263415a903192e740c6820d440b5d5571be060939c5f87285c9ee2ac17ddda4b072a
data/CHANGELOG.md CHANGED
@@ -1,5 +1,42 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.4.0 (2026-11-10)
4
+
5
+ **Upgrade from 0.3.0 if you train on MPS or CUDA.** 0.3.0 crashes when `evaluate` runs before `train`, and can run out of GPU memory during evaluation. Both are fixed here (see Fixed).
6
+
7
+ **This release changes LoRA defaults.** `LoraConfig#target_modules` now defaults to the query and value layers of the model's architecture, and targets match by name parts rather than substrings. See Changed.
8
+
9
+ ### Added
10
+
11
+ - **Datasets from files** (#30): `Dataset.from_jsonl(path)` and `Dataset.from_csv(path, **csv_options)`. A malformed JSONL line raises `ConfigurationError` naming the file and line.
12
+ - **`Dataset#split(test_size:, seed: 42)`** returns `[train, test]`, shuffled with `Random.new(seed)`. `test_size` is a fraction or a row count.
13
+ - **`Dataset#map` and `Dataset#filter`** return new Datasets, and `map` passes the block a copy of each row, so `row.delete(:label)` leaves the source Dataset unchanged. `map` returns an Array, as before, when the block doesn't return Hashes. `select` still returns an Array.
14
+ - **`Dataset#tokenize(tokenizer, column: :text, max_length: 512, keep: false)`** (#31) adds `input_ids` and `attention_mask` (and `token_type_ids` for text pairs), truncated to `max_length`. It takes a transformers-rb tokenizer or a `Tokenizers::Tokenizer`, and restores the latter's truncation setting afterwards.
15
+ - **`Trainers.from_pretrained(labels:)`** (#32) sets `num_labels`, `id2label` and `label2id`, so `config.json`, `load_pretrained` and Python see the class names. A `num_labels:` that contradicts `labels:` raises `ArgumentError`.
16
+ - **A log line when training starts** (#35) with the device, dtype, number of examples, epochs, optimizer steps, batch and accumulation, and how many parameters are trainable.
17
+ - `from_pretrained(task: :question_answering)` warns that transformers-rb 0.2.0 can't compute the question-answering loss.
18
+ - **`max_steps:`** (#34) trains for that many optimizer steps, overriding `num_train_epochs`. Epochs continue until the budget is spent, including when callbacks end some early with `should_epoch_stop`. The learning-rate schedule and `warmup_ratio` use it as the total.
19
+
20
+ ### Fixed
21
+
22
+ - **Token classification evaluates and predicts** (#27). `evaluate` crashed on per-token logits `[batch, seq, labels]`, and `predict` crashed when batches were padded to different lengths. Both now pad per-batch results to the longest batch with `-100` before joining them, so `compute_metrics` gets predictions and labels of matching shape. Training BERT for token classification also failed: transformers-rb 0.2.0's `BertForTokenClassification` raises `NameError` (`CrossEntropyLoss`) whenever it gets labels. The Trainer now runs it without labels and computes the loss itself.
23
+ - **Models whose `forward` takes no label keyword train and evaluate again.** `evaluate` passed the labels in and raised `ArgumentError: unknown keyword: :labels`. The Trainer now retries without them and computes the loss itself. Training gets the same fallback.
24
+ - **`evaluate` uses the model's own loss** when it returns one, instead of always recomputing cross-entropy over the logits. The fallback cross-entropy flattens per-token logits and ignores `-100` labels.
25
+ - **`eval_loss` is averaged over rows, not batches.** A short last batch used to count as much as a full one, so `eval_loss` can differ slightly from 0.3.0 when the eval set isn't a multiple of `per_device_eval_batch_size`.
26
+ - **CUDA is used when available** (#28). `resolved_device` only ever chose MPS or the CPU. The order is now an explicit `device:`, then CUDA, then MPS, then the CPU.
27
+ - **`DataCollatorWithPadding` truncates rows longer than `max_length`** (#29). Rows given as Arrays raised `ArgumentError: negative argument`. Rows given as Tensors were already truncated.
28
+ - **`DataCollatorWithPadding` pads Tensor labels with `-100`**, the same as Array labels. They used to be padded with `0`, a real class, so padding counted towards the loss.
29
+ - **Memory stays flat during training and evaluation.** torch-rb tensors are freed only when Ruby garbage-collects them, and Ruby can't see the memory LibTorch holds, so it rarely collects. Activations piled up batch after batch: evaluating DistilBERT on MPS ran out of memory at 22.6 GiB within 17 batches. The Trainer now runs a minor garbage collection after each batch, which kept that evaluation at 0.8 GB and costs under a millisecond per batch.
30
+ - **`evaluate` and `predict` move the model to the training device.** Only `train` did, so calling `evaluate` before `train` on MPS or CUDA, for a baseline, crashed with `Passed CPU tensor to MPS op`. Found by fine-tuning DistilBERT on MPS.
31
+ - `predict` no longer passes labels to the model, and puts the model back in training mode afterwards, as `evaluate` does.
32
+
33
+ ### Changed
34
+
35
+ - **LoRA `target_modules` defaults per architecture** (#33). It was `["query", "value"]`, which matched nothing on DistilBERT. The default is now `nil`, meaning `q_lin`/`v_lin` for DistilBERT, `query`/`value` for BERT and XLM-RoBERTa, `query_proj`/`value_proj` for DeBERTa-v2 and `q`/`v` for MPNet. Other transformers models need `target_modules:`, and models without a config keep `["query", "value"]`.
36
+ - **LoRA targets match whole name parts**, as in PEFT. `"query"` matches `attention.self.query` but no longer `pos_query_proj`. A target that only matches by substring still works, with a deprecation warning; substring matching ends in 0.5.0.
37
+ - **README "Support" section** (#35) replaces "Supported Tasks" and "Device Support". It has a table of architectures and tasks that the test suite checks against transformers-rb, and a platform matrix that separates tested in CI, hand-tested and unsupported. Question answering is now listed as not trainable. It was listed as supported before.
38
+ - New runtime dependency: `csv`, which is no longer a default gem from Ruby 3.4.
39
+
3
40
  ## 0.3.0 (2026-11-10)
4
41
 
5
42
  **This release changes LoRA training results.** `LoraModel.apply` now keeps the task head trainable, as PEFT does. It used to freeze it at its random initial weights. Expect better LoRA models than 0.2.0 with the same arguments. Full fine-tunes are unchanged.
@@ -19,6 +56,7 @@
19
56
  - **`save_model` writes the tokenizer for transformers-rb tokenizers** (#17). It used to save `tokenizer.json` only for objects that respond to `save`, which ruled out `AutoTokenizer` results such as `DistilBertTokenizerFast`, so no tokenizer was ever written. It now saves the backend as `tokenizer.json`, plus `tokenizer_config.json` with the tokenizer class and special tokens.
20
57
  - `model.safetensors` and `adapter_model.safetensors` carry the `"format" => "pt"` metadata that the transformers loaders in Ruby and Python check for.
21
58
 
59
+
22
60
  ### Changed
23
61
 
24
62
  - README: the LoRA example uses DistilBERT's module names (`q_lin`, `v_lin`) and shows its real output. The roadmap lists what is not planned, and "Known limitations" is gone, because its three issues are fixed.
@@ -27,6 +65,7 @@
27
65
 
28
66
  **This release changes training results.** Runs that use `gradient_accumulation_steps > 1` now follow the intended learning-rate schedule. All runs now apply `seed:`, so data order is different from 0.1.x and reproducible. Expect different (better) final models than 0.1.x with the same arguments.
29
67
 
68
+
30
69
  ### Changed
31
70
 
32
71
  - **A step is now one optimizer update** (#12). `global_step`, `max_steps`, `logging_steps`, `eval_steps`, `save_steps`, `warmup_steps` and the scheduler all count optimizer updates, not batches. The scheduler is sized as `ceil(batches_per_epoch / gradient_accumulation_steps) * num_train_epochs`, so it now decays to the end. Previously, with accumulation 4, a linear schedule ended near 75% of the peak learning rate.
@@ -58,6 +97,7 @@
58
97
  - `LoraModel.merge` no longer applies adapters twice. Merged layers kept adding the LoRA path on top of the merged weights, so a merged model's outputs were wrong. Each `LoraLinear` is now replaced by a plain `Torch::NN::Linear` holding the merged weights, so a saved merged model has the base model's parameter names. `LoraLinear#merge!` is idempotent and adds `#merged?` and `#to_linear`.
59
98
  - `Trainers::VERSION` now matches the gem version. The gemspec reads it from `lib/trainers/version.rb`. (#2)
60
99
 
100
+
61
101
  ### Changed
62
102
 
63
103
  - `TrainingArguments` validates its values at construction, and raises `ArgumentError` naming the argument, the value received and what was expected (#8):
data/README.md CHANGED
@@ -42,23 +42,19 @@ For Linux, and for running the tests, see [CONTRIBUTING.md](CONTRIBUTING.md).
42
42
  ```ruby
43
43
  require "trainers-rb"
44
44
 
45
- # Load a pre-trained model and tokenizer
45
+ # Load a pre-trained model and tokenizer. labels: names the classes in id
46
+ # order; they are saved in config.json.
46
47
  model, tokenizer = Trainers.from_pretrained(
47
48
  "distilbert-base-uncased",
48
49
  task: :sequence_classification,
49
- num_labels: 2
50
+ labels: %w[negative positive]
50
51
  )
51
52
 
52
- # Prepare your dataset
53
- train_data = texts.map.with_index do |text, i|
54
- encoded = tokenizer.(text, truncation: true, max_length: 128)
55
- {
56
- input_ids: encoded["input_ids"],
57
- attention_mask: encoded["attention_mask"],
58
- labels: labels[i]
59
- }
60
- end
61
- train_dataset = Trainers::Dataset.new(train_data)
53
+ # reviews.jsonl: {"text": "A gripping film.", "label": 1} on each line
54
+ dataset = Trainers::Dataset.from_jsonl("reviews.jsonl")
55
+ .map { |row| row.merge(labels: row.delete(:label)) }
56
+ .tokenize(tokenizer, column: :text, max_length: 128)
57
+ train_dataset, val_dataset = dataset.split(test_size: 0.1, seed: 42)
62
58
 
63
59
  # Configure and train
64
60
  args = Trainers::TrainingArguments.new(
@@ -122,7 +118,7 @@ config = Trainers::LoraConfig.new(
122
118
  r: 8, # rank
123
119
  lora_alpha: 16, # scaling factor
124
120
  lora_dropout: 0.1,
125
- target_modules: ["q_lin", "v_lin"], # which Linear layers to adapt (DistilBERT names)
121
+ # target_modules: ["q_lin", "v_lin"], # defaults to the attention query and value layers
126
122
  bias: :none, # :none, :all, or :lora_only
127
123
  task_type: :sequence_classification
128
124
  # modules_to_save: ["classifier"] # defaults to the task head for task_type
@@ -130,7 +126,7 @@ config = Trainers::LoraConfig.new(
130
126
 
131
127
  Trainers::LoraModel.apply(model, config)
132
128
  # => Training in full: pre_classifier, classifier
133
- # => LoRA applied to 12 modules: ...
129
+ # => LoRA target_modules ["q_lin", "v_lin"] (default) applied to 12 modules: ...
134
130
  # => trainable params: 739,586 || all params: 67,102,466 || trainable%: 1.1022%
135
131
 
136
132
  # Train as usual
@@ -154,6 +150,19 @@ The task head is trained in full, as in PEFT. `from_pretrained` initialises it r
154
150
 
155
151
  Names match the last part of a module's name. Pass `modules_to_save: []` to freeze the head, or your own list for other models.
156
152
 
153
+ `target_modules` defaults to the attention query and value layers of the model's `config.model_type`:
154
+
155
+ | `model_type` | Default `target_modules` |
156
+ |---|---|
157
+ | `bert`, `xlm-roberta` | `query`, `value` |
158
+ | `distilbert` | `q_lin`, `v_lin` |
159
+ | `deberta-v2` | `query_proj`, `value_proj` |
160
+ | `mpnet` | `q`, `v` |
161
+
162
+ Any other `model_type` needs `target_modules:`. Models without a transformers config default to `query` and `value`. Pass `:all_linear` to adapt every Linear layer.
163
+
164
+ A target matches a module when it is the module's full name or the last part of it, as in PEFT: `"query"` matches `attention.self.query` but not `pos_query_proj`. A target that matches nothing that way still matches by substring, with a deprecation warning. Substring matching ends in 0.5.0.
165
+
157
166
  ### Loading saved LoRA adapters
158
167
 
159
168
  ```ruby
@@ -195,6 +204,7 @@ Resume with the same `seed`, dataset and arguments as the original run. The trai
195
204
  |----------|---------|-------------|
196
205
  | `output_dir` | `"./output"` | Directory for checkpoints and saved models |
197
206
  | `num_train_epochs` | `3` | Number of training epochs |
207
+ | `max_steps` | `nil` | Number of optimizer steps to train for. Overrides `num_train_epochs` when set |
198
208
  | `per_device_train_batch_size` | `8` | Training batch size |
199
209
  | `per_device_eval_batch_size` | `8` | Evaluation batch size |
200
210
  | `learning_rate` | `5e-5` | Peak learning rate for AdamW |
@@ -330,16 +340,34 @@ args = Trainers::TrainingArguments.new(lr_scheduler_type: :constant, warmup_step
330
340
 
331
341
  ### Dataset
332
342
 
333
- Wrap an array of hashes:
343
+ Rows are Hashes with Symbol keys. Load them from a file, or wrap an Array:
344
+
345
+ ```ruby
346
+ dataset = Trainers::Dataset.from_jsonl("train.jsonl") # one JSON object per line
347
+ dataset = Trainers::Dataset.from_csv("train.csv") # header row gives the keys
348
+ dataset = Trainers::Dataset.from_csv("train.tsv", col_sep: "\t", converters: :numeric)
349
+ dataset = Trainers::Dataset.new([{ text: "A gripping film.", labels: 1 }])
350
+ ```
351
+
352
+ CSV values are Strings unless you pass `converters: :numeric`. Other options go to Ruby's `CSV`.
334
353
 
335
354
  ```ruby
336
- data = [
337
- { input_ids: [101, 2023, 2003], attention_mask: [1, 1, 1], labels: 1 },
338
- { input_ids: [101, 2919, 2143], attention_mask: [1, 1, 1], labels: 0 },
339
- ]
340
- dataset = Trainers::Dataset.new(data)
355
+ # A new Dataset with each row replaced. A block that doesn't return Hashes
356
+ # gives an Array, as Enumerable#map does.
357
+ dataset = dataset.map { |row| row.merge(labels: Integer(row[:label])) }
358
+ dataset = dataset.filter { |row| row[:text].size > 10 } # select returns an Array
359
+
360
+ # Shuffled with Random.new(seed): the same seed gives the same split
361
+ train, test = dataset.split(test_size: 0.2, seed: 42) # or a row count: test_size: 500
362
+
363
+ # input_ids and attention_mask, truncated to max_length tokens. The text
364
+ # column is dropped unless keep: true.
365
+ dataset = dataset.tokenize(tokenizer, column: :text, max_length: 256)
366
+ dataset = dataset.tokenize(tokenizer, column: [:premise, :hypothesis]) # text pairs
341
367
  ```
342
368
 
369
+ `tokenize` takes a transformers-rb tokenizer or a `Tokenizers::Tokenizer`.
370
+
343
371
  ### Data Collators
344
372
 
345
373
  Dynamic padding collator (pads each batch to the longest sequence in that batch):
@@ -354,32 +382,61 @@ Default collator (no padding, expects uniform-length inputs):
354
382
  collator = Trainers::DefaultDataCollator.new
355
383
  ```
356
384
 
357
- ## Supported Tasks
385
+ ## Support
386
+
387
+ ### Models and tasks
358
388
 
359
- trainers-rb works with any `Torch::NN::Module`. The `Trainers.from_pretrained` convenience method supports these transformers-rb model classes:
389
+ trainers-rb trains any `Torch::NN::Module` whose `forward` takes the batch's keys. `Trainers.from_pretrained` loads transformers-rb models through its `AutoModelFor*` classes, which cover these architectures:
360
390
 
361
- | Task | Model class |
362
- |------|-------------|
363
- | `:sequence_classification` | `AutoModelForSequenceClassification` |
364
- | `:token_classification` | `AutoModelForTokenClassification` |
365
- | `:question_answering` | `AutoModelForQuestionAnswering` |
391
+ <!-- architectures:start -->
392
+ | `model_type` | Sequence classification | Token classification | Question answering | Default LoRA targets |
393
+ |---|---|---|---|---|
394
+ | `bert` | ✓ | ✓ | – | `query`, `value` |
395
+ | `deberta-v2` | ✓ | – | – | `query_proj`, `value_proj` |
396
+ | `distilbert` | ✓ | – | ✗ no loss | `q_lin`, `v_lin` |
397
+ | `mpnet` | – | – | – | `q`, `v` |
398
+ | `xlm-roberta` | ✓ | – | – | `query`, `value` |
399
+ <!-- architectures:end -->
366
400
 
367
- You can also use any custom model:
401
+ This table is checked against transformers-rb by the test suite. **Question answering can't be trained yet:** transformers-rb 0.2.0's `DistilBertForQuestionAnswering` raises `Todo` when given `start_positions`, so there is no loss. transformers-rb also has heads that its `AutoModelFor*` classes don't reach, such as `MPNetForSequenceClassification`. Build those directly with their config and pass them to `Trainer`:
368
402
 
369
403
  ```ruby
370
404
  trainer = Trainers::Trainer.new(model: my_custom_model, args: args, ...)
371
405
  ```
372
406
 
373
- ## Device Support
407
+ ### Platforms
408
+
409
+ | | Tested in CI | Hand-tested | Not supported |
410
+ |---|---|---|---|
411
+ | Ruby | 3.3, 3.4, 4.0 | | 3.2 and older |
412
+ | torch-rb | 0.17.1, 0.27 | | older than 0.17.1 |
413
+ | OS | Ubuntu 24.04, macOS 15 (Apple Silicon) | macOS 27 | Windows (untested) |
414
+ | CPU | ✓ | ✓ | |
415
+ | MPS (Apple Silicon GPU) | | DistilBERT-base with LoRA, Apple M3 Pro | |
416
+ | CUDA (NVIDIA GPU) | | | Untested: chosen automatically, but nobody has run it yet |
417
+ | Multiple GPUs | | | Not planned |
418
+
419
+ Reports for other hardware are welcome as issues, with the "Training on" log line.
420
+
421
+ ### Devices
374
422
 
375
- trainers-rb auto-detects the best available device:
423
+ trainers-rb picks the first available device, in this order:
424
+
425
+ - **CUDA**: NVIDIA GPU
426
+ - **MPS**: Apple Silicon GPU, skipped with `no_mps: true`
427
+ - **CPU**: always available
428
+
429
+ The first log line of a run shows what it runs on. For DistilBERT with LoRA on an Apple M3 Pro:
430
+
431
+ ```
432
+ Training on mps (float32): 9,596 examples, 1 epoch, 600 optimizer steps (batch 16 x accumulation 1), 739,586 of 67,102,466 parameters trainable (1.10%)
433
+ ```
376
434
 
377
- - **CPU** — always available
378
- - **MPS** — Apple Silicon GPU, used automatically when available
435
+ That run, on 10,662 movie-review sentences ([rotten_tomatoes](https://www.cs.cornell.edu/people/pabo/movie-review-data/)) with 1,066 held out, reached 82.8% accuracy in 74 seconds. A full fine-tune reached 83.6% in 121 seconds.
379
436
 
380
437
  ```ruby
381
438
  # Force CPU
382
- args = Trainers::TrainingArguments.new(no_mps: true)
439
+ args = Trainers::TrainingArguments.new(device: Torch.device("cpu"))
383
440
 
384
441
  # Or set explicitly
385
442
  args = Trainers::TrainingArguments.new(device: Torch.device("mps"))
@@ -399,7 +456,7 @@ trainers-rb adds the training layer that transformers-rb intentionally omits. Bo
399
456
 
400
457
  ## Roadmap
401
458
 
402
- What comes next is chosen from real fine-tunes. If something was missing in yours, [open an issue](https://github.com/Selectus2/trainers-rb/issues): a report from a user outranks this list. Candidates are collected in [#20](https://github.com/Selectus2/trainers-rb/issues/20).
459
+ What comes next is chosen from real fine-tunes. If something was missing in yours, [open an issue](https://github.com/Selectus2/trainers-rb/issues): a report from a user outranks this list. Planned work is tracked in the [milestones](https://github.com/Selectus2/trainers-rb/milestones).
403
460
 
404
461
  **Not planned, at any version:** custom kernels, quantisation (including QLoRA), multi-GPU and distributed training, RL methods (GRPO, DPO), vision or speech models, GGUF export. torch-rb can't reach most of them from Ruby.
405
462
 
@@ -42,38 +42,22 @@ module Trainers
42
42
 
43
43
  private
44
44
 
45
+ # Pads, or truncates to max_length, each sequence of Integers
45
46
  def pad_and_stack(key, sequences)
46
- max_len = compute_max_length(sequences.map(&:length))
47
-
48
- pad_value = if key.to_s.include?("input_id")
49
- pad_token_id
50
- elsif key.to_s.include?("attention_mask")
51
- 0
52
- elsif key.to_s.include?("label")
53
- -100
54
- else
55
- 0
56
- end
47
+ max_len = compute_max_length(sequences.map(&:length))
48
+ pad_value = pad_value_for(key)
57
49
 
58
50
  padded = sequences.map do |seq|
59
- padded_seq = seq + [pad_value] * (max_len - seq.length)
60
- padded_seq
51
+ seq = seq.first(max_len)
52
+ seq + [pad_value] * (max_len - seq.length)
61
53
  end
62
54
 
63
- dtype = key.to_s.include?("attention_mask") ? :int64 : :int64
64
- Torch.tensor(padded, dtype: dtype)
55
+ Torch.tensor(padded, dtype: :int64)
65
56
  end
66
57
 
67
58
  def pad_and_stack_tensors(key, tensors)
68
- max_len = compute_max_length(tensors.map { |t| t.size(0) })
69
-
70
- pad_value = if key.to_s.include?("input_id")
71
- pad_token_id
72
- elsif key.to_s.include?("attention_mask")
73
- 0
74
- else
75
- 0
76
- end
59
+ max_len = compute_max_length(tensors.map { |t| t.size(0) })
60
+ pad_value = pad_value_for(key)
77
61
 
78
62
  padded = tensors.map do |t|
79
63
  if t.size(0) < max_len
@@ -87,6 +71,18 @@ module Trainers
87
71
  Torch.stack(padded)
88
72
  end
89
73
 
74
+ # Labels pad with -100, which cross-entropy ignores
75
+ def pad_value_for(key)
76
+ name = key.to_s
77
+ if name.include?("input_id")
78
+ pad_token_id
79
+ elsif name.include?("label")
80
+ -100
81
+ else
82
+ 0
83
+ end
84
+ end
85
+
90
86
  def compute_max_length(lengths)
91
87
  max_len = @max_length || lengths.max
92
88
  if @pad_to_multiple_of
@@ -1,11 +1,47 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "json"
4
+
3
5
  module Trainers
6
+ # Rows for training and evaluation: an Array of Hashes with Symbol keys,
7
+ # the shape the collators and Trainer expect.
8
+ #
9
+ # ds = Trainers::Dataset.from_jsonl("train.jsonl")
10
+ # train, eval = ds.split(test_size: 0.1)
11
+ # train = train.map { |row| row.merge(labels: Integer(row[:label])) }
4
12
  class Dataset
5
13
  include Enumerable
6
14
 
7
15
  attr_reader :data
8
16
 
17
+ # One JSON object per line. Blank lines are skipped.
18
+ def self.from_jsonl(path)
19
+ rows = []
20
+ File.foreach(path).with_index(1) do |line, number|
21
+ next if line.strip.empty?
22
+
23
+ row = begin
24
+ JSON.parse(line, symbolize_names: true)
25
+ rescue JSON::ParserError => e
26
+ raise ConfigurationError, "#{path}:#{number} is not valid JSON: #{e.message}"
27
+ end
28
+ raise ConfigurationError, "#{path}:#{number} is a JSON #{row.class}, not an object" unless row.is_a?(Hash)
29
+
30
+ rows << row
31
+ end
32
+ new(rows)
33
+ end
34
+
35
+ # A CSV file whose header row gives the keys. Values are Strings unless
36
+ # csv_options say otherwise, e.g. converters: :numeric. Other options
37
+ # (col_sep:, encoding: ...) go to CSV.
38
+ def self.from_csv(path, **csv_options)
39
+ require "csv"
40
+
41
+ table = CSV.read(path, headers: true, header_converters: :symbol, **csv_options)
42
+ new(table.map(&:to_h))
43
+ end
44
+
9
45
  def initialize(data)
10
46
  @data = data
11
47
  end
@@ -22,5 +58,118 @@ module Trainers
22
58
  def each(&block)
23
59
  @data.each(&block)
24
60
  end
61
+
62
+ # A new Dataset when the block returns a Hash for every row. Any other
63
+ # result gives an Array, as Enumerable#map does. The block gets a copy of
64
+ # each row, so row.delete(:label) leaves this Dataset unchanged.
65
+ def map(&block)
66
+ return enum_for(:map) unless block
67
+
68
+ results = @data.map { |row| block.call(row.is_a?(Hash) ? row.dup : row) }
69
+ results.all?(Hash) ? self.class.new(results) : results
70
+ end
71
+
72
+ # A new Dataset with the rows the block returns true for. select still
73
+ # returns an Array, as Enumerable#select does.
74
+ def filter(&block)
75
+ return enum_for(:filter) unless block
76
+
77
+ self.class.new(@data.select(&block))
78
+ end
79
+
80
+ TOKENIZE_BATCH = 1000
81
+
82
+ # A new Dataset with input_ids and attention_mask (and token_type_ids
83
+ # when the tokenizer gives them) for the text in column, truncated to
84
+ # max_length tokens. Pass two columns for text pairs, e.g. NLI. The text
85
+ # columns are dropped unless keep: true. Takes a Tokenizers::Tokenizer or
86
+ # a transformers-rb tokenizer.
87
+ def tokenize(tokenizer, column: :text, max_length: 512, keep: false)
88
+ columns = Array(column).map(&:to_sym)
89
+ unless columns.size.between?(1, 2)
90
+ raise ArgumentError, "column must name one text column, or two for text pairs, got #{column.inspect}"
91
+ end
92
+ unless max_length.is_a?(Integer) && max_length.positive?
93
+ raise ArgumentError, "max_length must be a positive Integer, got #{max_length.inspect}"
94
+ end
95
+
96
+ texts = @data.each_with_index.map do |row, index|
97
+ columns.map do |name|
98
+ text = row[name]
99
+ raise ConfigurationError, "Row #{index} has no String in #{name.inspect}: #{text.inspect}" unless text.is_a?(String)
100
+
101
+ text
102
+ end
103
+ end
104
+
105
+ rows = []
106
+ texts.each_slice(TOKENIZE_BATCH).with_index do |slice, n|
107
+ encoded = encode_texts(tokenizer, slice, max_length)
108
+ slice.each_index do |i|
109
+ row = @data[n * TOKENIZE_BATCH + i]
110
+ row = row.except(*columns) unless keep
111
+ rows << row.merge(encoded[i])
112
+ end
113
+ end
114
+ self.class.new(rows)
115
+ end
116
+
117
+ # Shuffles with Random.new(seed) and returns [train, test]. test_size is
118
+ # a fraction in (0, 1), or a row count.
119
+ def split(test_size:, seed: 42)
120
+ count = case test_size
121
+ when Float
122
+ raise ArgumentError, "test_size must be between 0 and 1, got #{test_size}" unless test_size > 0 && test_size < 1
123
+
124
+ (size * test_size).round
125
+ when Integer
126
+ test_size
127
+ else
128
+ raise ArgumentError, "test_size must be a Float in (0, 1) or an Integer row count, got #{test_size.inspect}"
129
+ end
130
+ unless count.between?(1, size - 1)
131
+ raise ArgumentError, "test_size #{test_size.inspect} leaves #{count} of #{size} rows for testing; " \
132
+ "both parts need at least one row"
133
+ end
134
+
135
+ rows = @data.shuffle(random: Random.new(seed))
136
+ [self.class.new(rows.drop(count)), self.class.new(rows.first(count))]
137
+ end
138
+
139
+ private
140
+
141
+ # One Hash of model inputs per text (or [text, pair])
142
+ def encode_texts(tokenizer, texts, max_length)
143
+ if tokenizer.respond_to?(:backend_tokenizer)
144
+ # transformers-rb tokenizer
145
+ first, pairs = texts.transpose
146
+ encoded = tokenizer.(first, text_pair: pairs, truncation: true, max_length: max_length)
147
+ keys = %i[input_ids attention_mask token_type_ids].select { |key| encoded[key] }
148
+ texts.each_index.map { |i| keys.to_h { |key| [key, encoded[key][i]] } }
149
+ elsif tokenizer.respond_to?(:encode_batch)
150
+ encode_with_backend(tokenizer, texts, max_length)
151
+ else
152
+ raise ConfigurationError, "#{tokenizer.class} is not a Tokenizers::Tokenizer or a transformers-rb tokenizer"
153
+ end
154
+ end
155
+
156
+ # Tokenizers::Tokenizer keeps truncation as a setting, so set it for the
157
+ # call and put the caller's back afterwards
158
+ def encode_with_backend(tokenizer, texts, max_length)
159
+ previous = tokenizer.truncation
160
+ tokenizer.enable_truncation(max_length)
161
+ inputs = texts.map { |text| text.size == 1 ? text.first : text }
162
+ tokenizer.encode_batch(inputs).map do |encoding|
163
+ row = { input_ids: encoding.ids, attention_mask: encoding.attention_mask }
164
+ row[:token_type_ids] = encoding.type_ids if texts.first.size == 2
165
+ row
166
+ end
167
+ ensure
168
+ tokenizer.no_truncation
169
+ if previous
170
+ options = previous.reject { |key, _| key == "max_length" }.transform_keys(&:to_sym)
171
+ tokenizer.enable_truncation(previous["max_length"], **options)
172
+ end
173
+ end
25
174
  end
26
175
  end
@@ -9,12 +9,25 @@ module Trainers
9
9
  r: 8,
10
10
  lora_alpha: 16,
11
11
  lora_dropout: 0.0,
12
- target_modules: ["query", "value"], # or :all_linear
12
+ target_modules: nil, # nil: DEFAULT_TARGETS for the model; or names, or :all_linear
13
13
  bias: :none, # :none, :all, :lora_only
14
14
  task_type: :sequence_classification,
15
15
  modules_to_save: nil # nil: the task head for task_type
16
16
  }.freeze
17
17
 
18
+ # Attention query and value projections for each transformers-rb
19
+ # model_type, used when target_modules is nil
20
+ DEFAULT_TARGETS = {
21
+ "bert" => %w[query value],
22
+ "xlm-roberta" => %w[query value],
23
+ "distilbert" => %w[q_lin v_lin],
24
+ "deberta-v2" => %w[query_proj value_proj],
25
+ "mpnet" => %w[q v]
26
+ }.freeze
27
+
28
+ # Targets for models without a transformers config, the default before 0.4.0
29
+ FALLBACK_TARGETS = %w[query value].freeze
30
+
18
31
  # Modules that stay fully trainable for each task_type, so the task head
19
32
  # (randomly initialised by from_pretrained) learns along with the adapters
20
33
  TASK_HEADS = {
@@ -36,6 +49,24 @@ module Trainers
36
49
  Array(@modules_to_save.nil? ? TASK_HEADS.fetch(@task_type&.to_sym, []) : @modules_to_save).map(&:to_s)
37
50
  end
38
51
 
52
+ # target_modules when given, otherwise the defaults for the model's
53
+ # config.model_type, or FALLBACK_TARGETS for a model without a config.
54
+ # Returns [targets, source], source being :given, :model_type or :fallback.
55
+ def resolved_target_modules(model)
56
+ return [@target_modules, :given] unless @target_modules.nil?
57
+
58
+ config = model.respond_to?(:config) ? model.config : nil
59
+ return [FALLBACK_TARGETS, :fallback] unless config.respond_to?(:model_type)
60
+
61
+ type = config.model_type.to_s
62
+ targets = DEFAULT_TARGETS[type]
63
+ unless targets
64
+ raise AdapterError, "No default target_modules for model_type #{type.inspect}. Pass target_modules: with " \
65
+ "the Linear modules to adapt, from: #{LoraUtils.linear_names(model).first(20).join(', ')}"
66
+ end
67
+ [targets, :model_type]
68
+ end
69
+
39
70
  def scaling
40
71
  @lora_alpha.to_f / @r
41
72
  end
@@ -18,11 +18,13 @@ module Trainers
18
18
  model.parameters.each { |p| p.requires_grad = false }
19
19
 
20
20
  # Find and replace target modules
21
- targets = LoraUtils.find_target_modules(model, config.target_modules)
21
+ # The per-architecture defaults are exact names, so they never fall back to substrings
22
+ target_modules, source = config.resolved_target_modules(model)
23
+ targets = LoraUtils.find_target_modules(model, target_modules, fallback: source != :model_type)
22
24
 
23
25
  if targets.empty?
24
- raise AdapterError, "No Linear modules found matching target_modules: #{config.target_modules.inspect}. " \
25
- "Available modules: #{model.named_modules.map(&:first).join(', ')}"
26
+ raise AdapterError, "No Linear modules found matching target_modules: #{target_modules.inspect}. " \
27
+ "Linear modules: #{LoraUtils.linear_names(model).first(20).join(', ')}"
26
28
  end
27
29
 
28
30
  targets.each do |name, linear|
@@ -53,7 +55,8 @@ module Trainers
53
55
 
54
56
  unfreeze_modules_to_save(model, config)
55
57
 
56
- Trainers.log(:info, "LoRA applied to #{targets.size} modules: #{targets.keys.join(', ')}")
58
+ Trainers.log(:info, "LoRA target_modules #{target_modules.inspect}#{' (default)' unless source == :given} " \
59
+ "applied to #{targets.size} modules: #{targets.keys.join(', ')}")
57
60
  LoraUtils.print_trainable_parameters(model)
58
61
 
59
62
  model
@@ -2,23 +2,36 @@
2
2
 
3
3
  module Trainers
4
4
  module LoraUtils
5
- # Find all Linear modules in a model matching the target pattern
6
- def self.find_target_modules(model, target_modules)
7
- targets = {}
8
-
9
- model.named_modules.each do |name, mod|
10
- next unless mod.is_a?(Torch::NN::Linear)
5
+ # Find the Linear modules matching target_modules, as PEFT does: a name
6
+ # matches a target when it is the target or ends with ".target". A target
7
+ # that matches nothing that way falls back to matching any name that
8
+ # contains it, with a deprecation warning; substring matching ends in 0.5.0.
9
+ # fallback: false turns the fallback off (for the per-architecture defaults).
10
+ def self.find_target_modules(model, target_modules, fallback: true)
11
+ linears = model.named_modules.select { |_, mod| mod.is_a?(Torch::NN::Linear) }
12
+ return linears.to_h if target_modules == :all_linear
11
13
 
12
- if target_modules == :all_linear
13
- targets[name] = mod
14
- elsif target_modules.is_a?(Array)
15
- if target_modules.any? { |pattern| name.include?(pattern) }
16
- targets[name] = mod
14
+ targets = {}
15
+ Array(target_modules).map(&:to_s).each do |pattern|
16
+ matched = linears.select { |name, _| name == pattern || name.end_with?(".#{pattern}") }
17
+ if matched.empty? && fallback
18
+ matched = linears.select { |name, _| name.include?(pattern) }
19
+ unless matched.empty?
20
+ Trainers.log(:warn, "target_modules #{pattern.inspect} matched #{matched.map(&:first).join(', ')} by substring. " \
21
+ "Substring matching is deprecated and ends in 0.5.0; name the modules by their last part " \
22
+ "or full name instead.")
17
23
  end
18
24
  end
25
+ matched.each { |name, mod| targets[name] = mod }
19
26
  end
20
27
 
21
- targets
28
+ # Keep the model's module order
29
+ linears.select { |name, _| targets.key?(name) }.to_h
30
+ end
31
+
32
+ # Names of the Linear modules, for error messages
33
+ def self.linear_names(model)
34
+ model.named_modules.select { |_, mod| mod.is_a?(Torch::NN::Linear) }.map(&:first)
22
35
  end
23
36
 
24
37
  # Replace a child module on a parent by setting the instance variable.
@@ -46,20 +46,31 @@ module Trainers
46
46
  # One step is one optimizer update. The last accumulation window of an
47
47
  # epoch may be shorter than gradient_accumulation_steps; it still steps.
48
48
  steps_per_epoch = (batches_per_epoch.to_f / accum).ceil
49
- total_steps = steps_per_epoch * @args.num_train_epochs
49
+ if @args.max_steps
50
+ total_steps = @args.max_steps
51
+ num_epochs = (total_steps.to_f / steps_per_epoch).ceil
52
+ else
53
+ num_epochs = @args.num_train_epochs
54
+ total_steps = steps_per_epoch * num_epochs
55
+ end
50
56
 
51
57
  @state.max_steps = total_steps
52
- @state.num_train_epochs = @args.num_train_epochs
58
+ @state.num_train_epochs = num_epochs
53
59
 
54
60
  @optimizer = create_optimizer
55
61
  @lr_scheduler = create_scheduler(total_steps)
56
62
  @optimizer.zero_grad
63
+ log_training_setup(device, num_epochs, total_steps)
57
64
  start_epoch, skip_batches = resume_from_checkpoint
58
65
 
59
66
  @callback_handler.fire(:on_train_begin, @args, @state, @control)
60
67
 
61
- @args.num_train_epochs.times do |epoch|
68
+ # With max_steps, epochs continue until the step budget is spent, even
69
+ # when callbacks end some epochs early with should_epoch_stop
70
+ (0..).each do |epoch|
71
+ break if @args.max_steps.nil? && epoch >= num_epochs
62
72
  next if epoch < start_epoch
73
+ break if @state.global_step >= total_steps # done, or resumed from the final checkpoint
63
74
 
64
75
  @state.epoch = epoch + 1
65
76
  @callback_handler.fire(:on_epoch_begin, @args, @state, @control)
@@ -87,6 +98,7 @@ module Trainers
87
98
 
88
99
  epoch_loss += loss.item
89
100
  epoch_batches += 1
101
+ release_memory
90
102
 
91
103
  next unless position == window_size - 1
92
104
 
@@ -95,6 +107,7 @@ module Trainers
95
107
  @lr_scheduler.step
96
108
  @optimizer.zero_grad
97
109
  @state.global_step += 1
110
+ @control.should_training_stop = true if @state.global_step >= total_steps
98
111
 
99
112
  # Logging
100
113
  if should_log?
@@ -167,42 +180,39 @@ module Trainers
167
180
  raise ArgumentError, "No eval_dataset provided" unless dataset
168
181
 
169
182
  device = @args.resolved_device
183
+ @model.to(device)
170
184
  @model.eval
171
185
 
172
186
  all_preds = []
173
187
  all_labels = []
174
188
  total_loss = 0.0
175
- total_steps = 0
189
+ examples = 0
176
190
 
177
191
  Torch.no_grad do
178
192
  each_batch(dataset, @args.per_device_eval_batch_size) do |batch|
179
- batch = move_to_device(batch, device)
180
- labels = pop_labels(batch).values.first
181
-
182
- output = forward(batch)
193
+ batch = move_to_device(batch, device)
194
+ output, labels = forward_with_labels(batch)
195
+ logits = logits_of(output)
183
196
 
184
197
  if labels
185
- logits = output.respond_to?(:logits) ? output.logits : output
186
- loss = Torch::NN::F.cross_entropy(logits, labels)
187
- total_loss += loss.item
198
+ # Weight by batch size, so a short last batch counts for its rows only
199
+ size = labels.size(0)
200
+ total_loss += loss_for(output, logits, labels).item * size
201
+ examples += size
188
202
  all_labels << labels.detach.cpu
189
203
  end
190
- total_steps += 1
191
-
192
- logits = output.respond_to?(:logits) ? output.logits : output
193
204
  all_preds << logits.detach.cpu
205
+ release_memory
194
206
  end
195
207
  end
196
208
 
197
209
  @model.train
198
210
 
199
211
  metrics = {}
200
- metrics[:eval_loss] = total_loss / total_steps if total_steps > 0
212
+ metrics[:eval_loss] = total_loss / examples if examples > 0
201
213
 
202
214
  if @compute_metrics && all_preds.any? && all_labels.any?
203
- preds = Torch.cat(all_preds)
204
- labels = Torch.cat(all_labels)
205
- eval_pred = EvalPrediction.new(predictions: preds, label_ids: labels)
215
+ eval_pred = EvalPrediction.new(predictions: concat_padded(all_preds), label_ids: concat_padded(all_labels))
206
216
  custom_metrics = @compute_metrics.call(eval_pred)
207
217
  metrics.merge!(custom_metrics)
208
218
  end
@@ -210,21 +220,25 @@ module Trainers
210
220
  metrics
211
221
  end
212
222
 
223
+ # Logits for every row. Batches padded to different lengths, as in token
224
+ # classification, are padded to the longest with -100.
213
225
  def predict(test_dataset)
214
226
  device = @args.resolved_device
227
+ @model.to(device)
215
228
  @model.eval
216
229
 
217
230
  all_preds = []
218
231
  Torch.no_grad do
219
232
  each_batch(test_dataset, @args.per_device_eval_batch_size) do |batch|
220
- batch = move_to_device(batch, device)
221
- output = forward(batch)
222
- logits = output.respond_to?(:logits) ? output.logits : output
223
- all_preds << logits.detach.cpu
233
+ batch = move_to_device(batch, device)
234
+ pop_labels(batch)
235
+ all_preds << logits_of(forward(batch)).detach.cpu
236
+ release_memory
224
237
  end
225
238
  end
239
+ @model.train
226
240
 
227
- Torch.cat(all_preds)
241
+ concat_padded(all_preds)
228
242
  end
229
243
 
230
244
  def save_model(output_dir = nil)
@@ -234,35 +248,103 @@ module Trainers
234
248
 
235
249
  private
236
250
 
251
+ # One line with what is about to run, so a run on the wrong device or
252
+ # with the wrong number of trainable weights shows up before it starts
253
+ def log_training_setup(device, num_epochs, total_steps)
254
+ params = @model.parameters
255
+ total = params.sum(&:numel)
256
+ trainable = params.select(&:requires_grad).sum(&:numel)
257
+ dtype = params.first&.dtype || "no parameters"
258
+ percent = total.positive? ? trainable * 100.0 / total : 0.0
259
+
260
+ Trainers.log(:info, "Training on #{device.respond_to?(:type) ? device.type : device} (#{dtype}): #{LoraUtils.format_number(@train_dataset.size)} examples, " \
261
+ "#{num_epochs} epoch#{'s' unless num_epochs == 1}, #{LoraUtils.format_number(total_steps)} optimizer steps " \
262
+ "(batch #{@args.per_device_train_batch_size} x accumulation #{@args.gradient_accumulation_steps}), " \
263
+ "#{LoraUtils.format_number(trainable)} of #{LoraUtils.format_number(total)} " \
264
+ "parameters trainable (#{format('%.2f', percent)}%)")
265
+ end
266
+
237
267
  def compute_loss(batch)
268
+ output, labels = forward_with_labels(batch)
269
+ if labels.nil? && !(output.respond_to?(:loss) && output.loss)
270
+ raise ConfigurationError, "Model did not return a loss and no labels found in batch. " \
271
+ "Either pass labels in your dataset or use a model that computes loss."
272
+ end
273
+
274
+ loss_for(output, logits_of(output), labels)
275
+ end
276
+
277
+ # Runs the model with the labels, so models that compute their own loss
278
+ # return it. Returns [output, labels]; labels is nil when the batch has none.
279
+ def forward_with_labels(batch)
238
280
  label_values = pop_labels(batch)
239
281
  labels = label_values.values.first
240
282
 
241
- # Try passing labels to the model (some models compute loss internally)
242
283
  output = begin
243
284
  forward(batch.merge(label_values))
285
+ rescue NameError => e
286
+ # transformers-rb 0.2.0's BertForTokenClassification computes its loss
287
+ # with an unqualified CrossEntropyLoss and raises whenever it gets labels
288
+ raise unless e.name == :CrossEntropyLoss
289
+
290
+ forward(batch)
291
+ rescue ArgumentError => e
292
+ # A model whose forward takes no label keyword: run it without labels
293
+ # and compute the loss outside it
294
+ raise unless e.message.match?(/unknown keywords?:/) && label_values.keys.any? { |key| e.message.include?(":#{key}") }
295
+
296
+ forward(batch)
244
297
  rescue => e
245
298
  # If the model doesn't support labels kwarg (e.g. transformers-rb Todo),
246
299
  # fall back to forward without labels + external loss
247
- if e.message.include?("Todo") || e.message.include?("not implemented")
248
- forward(batch)
249
- else
250
- raise
251
- end
300
+ raise unless e.message.include?("Todo") || e.message.include?("not implemented")
301
+
302
+ forward(batch)
252
303
  end
253
304
 
254
- # Restore labels to batch for downstream use
255
- batch.merge!(label_values)
305
+ [output, labels]
306
+ end
256
307
 
257
- if output.respond_to?(:loss) && output.loss
258
- output.loss
259
- elsif labels
260
- logits = output.respond_to?(:logits) ? output.logits : output
261
- Torch::NN::F.cross_entropy(logits, labels)
308
+ # The model's own loss when it returns one, otherwise cross-entropy over
309
+ # the logits. Per-token logits [batch, seq, classes] are flattened, and
310
+ # labels of -100 (padding) are ignored.
311
+ def loss_for(output, logits, labels)
312
+ return output.loss if output.respond_to?(:loss) && output.loss
313
+
314
+ if logits.dim > 2
315
+ Torch::NN::F.cross_entropy(logits.reshape(-1, logits.size(-1)), labels.reshape(-1))
262
316
  else
263
- raise ConfigurationError, "Model did not return a loss and no labels found in batch. " \
264
- "Either pass labels in your dataset or use a model that computes loss."
317
+ Torch::NN::F.cross_entropy(logits, labels)
318
+ end
319
+ end
320
+
321
+ # torch-rb tensors are freed when Ruby garbage-collects them, and Ruby
322
+ # cannot see the memory LibTorch holds for them, so it rarely collects.
323
+ # Each batch's activations would then pile up: evaluating DistilBERT on
324
+ # MPS reached 22 GiB in 17 batches. A minor collection per batch keeps
325
+ # memory flat and costs under a millisecond.
326
+ def release_memory
327
+ GC.start(full_mark: false, immediate_sweep: true)
328
+ end
329
+
330
+ def logits_of(output)
331
+ output.respond_to?(:logits) ? output.logits : output
332
+ end
333
+
334
+ # Concatenates per-batch tensors along dim 0, first padding dim 1 to the
335
+ # longest batch with -100
336
+ def concat_padded(tensors)
337
+ return Torch.cat(tensors) if tensors.first.dim < 2
338
+
339
+ length = tensors.map { |t| t.size(1) }.max
340
+ padded = tensors.map do |t|
341
+ next t if t.size(1) == length
342
+
343
+ shape = t.shape.dup
344
+ shape[1] = length - t.size(1)
345
+ Torch.cat([t, Torch.full(shape, -100, dtype: t.dtype)], 1)
265
346
  end
347
+ Torch.cat(padded)
266
348
  end
267
349
 
268
350
  def forward(batch)
@@ -5,6 +5,7 @@ module Trainers
5
5
  DEFAULTS = {
6
6
  output_dir: "./output",
7
7
  num_train_epochs: 3,
8
+ max_steps: nil, # optimizer steps; overrides num_train_epochs when set
8
9
  per_device_train_batch_size: 8,
9
10
  per_device_eval_batch_size: 8,
10
11
  learning_rate: 5e-5,
@@ -56,10 +57,14 @@ module Trainers
56
57
  validate!
57
58
  end
58
59
 
60
+ # The device training runs on: device: when given, otherwise CUDA, then
61
+ # MPS (unless no_mps:), then the CPU
59
62
  def resolved_device
60
63
  return @device if @device
61
64
 
62
- if !@no_mps && defined?(Torch::Backends::MPS) && Torch::Backends::MPS.available?
65
+ if Torch::CUDA.available?
66
+ Torch.device("cuda")
67
+ elsif !@no_mps && defined?(Torch::Backends::MPS) && Torch::Backends::MPS.available?
63
68
  Torch.device("mps")
64
69
  else
65
70
  Torch.device("cpu")
@@ -88,6 +93,7 @@ module Trainers
88
93
  end
89
94
  check(:warmup_steps, "a non-negative Integer") { |v| v.is_a?(Integer) && !v.negative? }
90
95
  check(:warmup_ratio, "a number between 0 and 1") { |v| v.is_a?(Numeric) && v.between?(0, 1) }
96
+ check(:max_steps, "nil or a positive Integer") { |v| v.nil? || (v.is_a?(Integer) && v.positive?) }
91
97
  check(:eval_steps, "nil or a positive Integer") { |v| v.nil? || (v.is_a?(Integer) && v.positive?) }
92
98
  check(:seed, "nil or a non-negative Integer") { |v| v.nil? || (v.is_a?(Integer) && !v.negative?) }
93
99
  check(:resume_from_checkpoint, "nil, true or a checkpoint directory String") { |v| v.nil? || v == true || v.is_a?(String) }
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Trainers
4
- VERSION = "0.3.0"
4
+ VERSION = "0.4.0"
5
5
  end
data/lib/trainers.rb CHANGED
@@ -31,10 +31,15 @@ module Trainers
31
31
  seed
32
32
  end
33
33
 
34
- # Convenience method: load model + tokenizer and prepare for training
35
- def self.from_pretrained(model_name, task: :sequence_classification, num_labels: 2)
34
+ # Convenience method: load model + tokenizer and prepare for training.
35
+ # labels: names the classes, in id order, so the saved config.json carries
36
+ # id2label and label2id; it sets num_labels to labels.size.
37
+ def self.from_pretrained(model_name, task: :sequence_classification, num_labels: nil, labels: nil)
36
38
  require "transformers-rb"
37
39
 
40
+ labels = label_names(labels, num_labels)
41
+ num_labels ||= labels ? labels.size : 2
42
+
38
43
  model_class = case task
39
44
  when :sequence_classification
40
45
  Transformers::AutoModelForSequenceClassification
@@ -46,12 +51,39 @@ module Trainers
46
51
  Transformers::AutoModel
47
52
  end
48
53
 
54
+ if task == :question_answering
55
+ Trainers.log(:warn, "transformers-rb #{Transformers::VERSION} cannot compute the question-answering loss " \
56
+ "(DistilBertForQuestionAnswering raises Todo when given start_positions), so this model " \
57
+ "can predict but not be trained")
58
+ end
59
+
49
60
  model = model_class.from_pretrained(model_name, num_labels: num_labels)
50
61
  tokenizer = Transformers::AutoTokenizer.from_pretrained(model_name)
51
62
 
63
+ # transformers-rb 0.2.0 config has no id2label or label2id writer, and
64
+ # from_pretrained drops them when passed as options
65
+ if labels
66
+ model.config.instance_variable_set(:@id2label, labels.each_with_index.to_h { |name, id| [id, name] })
67
+ model.config.instance_variable_set(:@label2id, labels.each_with_index.to_h)
68
+ end
69
+
52
70
  [model, tokenizer]
53
71
  end
54
72
 
73
+ # Internal: labels as unique Strings, checked against num_labels
74
+ def self.label_names(labels, num_labels)
75
+ return if labels.nil?
76
+
77
+ names = Array(labels).map(&:to_s)
78
+ raise ArgumentError, "labels must be a non-empty list of unique names, got #{labels.inspect}" if names.empty? || names.uniq.size != names.size
79
+ if num_labels && num_labels != names.size
80
+ raise ArgumentError, "num_labels: #{num_labels} contradicts labels: #{names.inspect} (#{names.size} labels)"
81
+ end
82
+
83
+ names
84
+ end
85
+ private_class_method :label_names
86
+
55
87
  # Loads a directory written by Trainer#save_model. Returns [model, tokenizer].
56
88
  def self.load_pretrained(model_dir)
57
89
  SaveUtils.load_pretrained(model_dir)
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: trainers-rb
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.0
4
+ version: 0.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Vishwajeetsingh Desurkar
@@ -80,6 +80,20 @@ dependencies:
80
80
  - - ">="
81
81
  - !ruby/object:Gem::Version
82
82
  version: '1.5'
83
+ - !ruby/object:Gem::Dependency
84
+ name: csv
85
+ requirement: !ruby/object:Gem::Requirement
86
+ requirements:
87
+ - - ">="
88
+ - !ruby/object:Gem::Version
89
+ version: '3.2'
90
+ type: :runtime
91
+ prerelease: false
92
+ version_requirements: !ruby/object:Gem::Requirement
93
+ requirements:
94
+ - - ">="
95
+ - !ruby/object:Gem::Version
96
+ version: '3.2'
83
97
  - !ruby/object:Gem::Dependency
84
98
  name: rake
85
99
  requirement: !ruby/object:Gem::Requirement