rpeft 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d6162af3628a1e82e72c85f6f1ebf42c1f4bd58298992d4bc7036766628595c4
4
- data.tar.gz: 52b2ac5be8272a31978e751e6d388e010cc97625b8cabeea4f6f6690bad3cb52
3
+ metadata.gz: e72e65f59460cc4217035e8043cf25d6e2b1514638fd20589cdcf6949a9508ab
4
+ data.tar.gz: aeacf67c2210d3dc50508e21741badf255015cdcabcde184e8c51d1f83433b6f
5
5
  SHA512:
6
- metadata.gz: ac75ae251f4f228ee5e3fca1b104f7819d509c10644754fb15754ade5bc82d03af67e3619c1492c9fbf715ca26f82e3e673caaf493f46114fd100271e28760bc
7
- data.tar.gz: a3a06a0a30dd6419f92180c4ce3fa12b20f1a0a0ae45e503f0cd9b25afd23611f3bb1ed580f7e60899ab42c4d05ec79f9d317919e01b12aa00cc5aaa8f5c3e5f
6
+ metadata.gz: 400c1b46c76250c1073488c1856967e6581b6cab2378e305208ca246ce312b4eac1066f2d9b1d1d8a2bc64bb1855f63650a516ea7af5f0aa102836d51e9545dc
7
+ data.tar.gz: bf4a220e57f61852471ad93319576fdab39ed06936455b5c20565f7048dd0becd3b88f49d9567957ec1566a3552b50313acc4fa000180292c2ea3cfc25a67356
data/README.md CHANGED
@@ -12,6 +12,7 @@ It is an independent project, not affiliated with Hugging Face.
12
12
  - LoRA fine-tuning from JSONL files or Ruby Enumerables, in plain text, instruction, and chat formats
13
13
  - Merging an adapter into its base model to produce a standalone model
14
14
  - Generation and chat with a merged model, a Hugging Face model id, or an adapter on top of its base model
15
+ - GGUF export of a merged model for llama.cpp, Ollama, and other local runtimes
15
16
  - Apple Silicon (MPS) as a first-class target, with CUDA and CPU auto-detected
16
17
  - Validation loss, per-epoch checkpoints, and resuming
17
18
  - A CLI for training, merging, and generation, and `rpeft doctor` to check the Python environment
@@ -248,6 +249,59 @@ Apple Silicon Macs (M1 and later) are a first-class target: small models such as
248
249
  Training uses float32 on MPS (half precision there is numerically fragile), bfloat16 on capable CUDA GPUs, and `PYTORCH_ENABLE_MPS_FALLBACK=1` is set automatically for
249
250
  operators MPS does not implement. Merging always runs on CPU, which is the memory-safest place for it.
250
251
 
252
+ ### Exporting to GGUF
253
+
254
+ A merged model can be written to a single GGUF file for llama.cpp, Ollama, and other local runtimes.
255
+ The conversion is done by llama.cpp's own `convert_hf_to_gguf.py`, which is not published as a package, so RPEFT needs a llama.cpp checkout:
256
+
257
+ ```console
258
+ $ git clone https://github.com/ggml-org/llama.cpp ~/src/llama.cpp
259
+ $ .venv/bin/pip install sentencepiece # or, with uv: uv pip install sentencepiece
260
+ ```
261
+
262
+ ```ruby
263
+ model = adapter.merge(output_dir: './ruby-model')
264
+ model.export_gguf('./ruby-model.gguf', llama_cpp: '~/src/llama.cpp') # => "/path/to/ruby-model.gguf"
265
+ ```
266
+
267
+ `llama_cpp:` defaults to the `LLAMA_CPP_DIR` environment variable. The converter runs in its own process with the Python RPEFT uses,
268
+ so the packages it needs beyond RPEFT's own go into the same environment. `sentencepiece` is one of them even for models without
269
+ a SentencePiece tokenizer, such as Qwen2.5, because the converter imports it before looking for one.
270
+
271
+ `quantize:` picks the type of the weights. `:q8_0` (the default), `:f16`, and `:bf16` are written by the converter directly.
272
+ `:q6_k`, `:q5_k_m`, and `:q4_k_m` are smaller and also need `llama-quantize`, found on `PATH` (`brew install llama.cpp`), in the checkout's `build/bin`,
273
+ or given as `llama_quantize:`. Small models lose more to quantization, which is why the default stays at 8 bits.
274
+
275
+ Export a model you merged yourself, from a directory no one else writes to while the export runs; an adapter is merged first.
276
+ An existing output file is never overwritten, and a failed export leaves nothing behind.
277
+
278
+ A GGUF file carries the model's chat template, and Ollama (checked with 0.33.1) wraps every prompt in it, even when a Modelfile gives a `TEMPLATE` of its own.
279
+ A model fine-tuned on plain text, like the example above, expects the prompt as it was in training, so export it without the chat template:
280
+
281
+ ```ruby
282
+ model.export_gguf('./ruby-model.gguf', chat_template: false, llama_cpp: '~/src/llama.cpp')
283
+ ```
284
+
285
+ This removes the default chat template, the one Ollama used in that check, and the list of named ones; templates stored under their own names, such as one for tool use, stay.
286
+ Leaving it out uses `gguf_new_metadata.py` from the same checkout and briefly needs room for one more copy of the file.
287
+ Keep the chat template for a model fine-tuned on chat-format data: Ollama then needs a Modelfile with the `FROM` line alone, and its chat answers as `#chat` does.
288
+
289
+ To run a file exported without its chat template in Ollama, point a Modelfile at it and give a template that passes the prompt through:
290
+
291
+ ```text
292
+ FROM ./ruby-model.gguf
293
+ TEMPLATE {{ .Prompt }}
294
+ ```
295
+
296
+ ```console
297
+ $ ollama create ruby-model -f Modelfile
298
+ $ ollama run ruby-model $'User: How do I find the minimum value in an array?\nAssistant:'
299
+ ```
300
+
301
+ The converter loads some tokenizers with `trust_remote_code=True`, so it runs offline with empty Hugging Face caches of its own, and a model directory is refused when it holds Python code,
302
+ `auto_map` in a JSON file, a link leading out of it or to a directory, a hard link, or a special file. Any of these could run code or carry another file, such as a credential, into the GGUF file.
303
+ The checks run before the converter starts, which is why the directory must not change during the export.
304
+
251
305
  ### Environment check
252
306
 
253
307
  ```ruby
@@ -256,7 +310,7 @@ RPEFT.check # => a Hash describing Python, packages, and devices
256
310
 
257
311
  ```console
258
312
  $ rpeft doctor
259
- Ruby: 4.0.6 (rpeft 0.2.0)
313
+ Ruby: 4.0.9 (rpeft 0.3.0)
260
314
  Python: 3.12.13 (/path/to/.venv/bin/python)
261
315
  torch: OK 2.13.0
262
316
  transformers: OK 5.15.0
@@ -264,18 +318,23 @@ peft: OK 0.20.0
264
318
  datasets: OK 5.0.1
265
319
  accelerate: OK 1.14.0
266
320
  Device: mps (CUDA: no, MPS: yes)
321
+ llama.cpp: OK /Users/you/src/llama.cpp
322
+ Quantizer: OK /opt/homebrew/bin/llama-quantize
267
323
  Status: OK
268
324
  ```
269
325
 
326
+ The llama.cpp lines are optional, for GGUF export only, and never affect the status.
327
+
270
328
  ### Errors
271
329
 
272
330
  Python exceptions never surface raw. They are translated into `RPEFT::DependencyError`, `RPEFT::ModelLoadError`, `RPEFT::DatasetError`, `RPEFT::TrainingError`,
273
331
  `RPEFT::MergeError`, or `RPEFT::GenerationError`, and the original Python error (with its traceback) stays reachable via `error.cause`.
332
+ A failing llama.cpp tool during GGUF export raises `RPEFT::ExportError` with the end of the tool's output.
274
333
  Invalid arguments are rejected in Ruby with `RPEFT::ConfigurationError` before the backend is called.
275
334
 
276
335
  ## CLI
277
336
 
278
- The CLI covers the core of the lifecycle: training, merging, and generation from a prompt.
337
+ The CLI covers the core of the lifecycle: training, merging, generation from a prompt, and GGUF export.
279
338
  Validation datasets, checkpoints, and chat are available from the Ruby API only.
280
339
 
281
340
  ```console
@@ -311,6 +370,16 @@ $ rpeft generate \
311
370
  --prompt $'User: How do I find the minimum value in an array?\nAssistant:'
312
371
  ```
313
372
 
373
+ A merged model is exported to GGUF with `rpeft export gguf`, which takes the same choices as `export_gguf`:
374
+
375
+ ```console
376
+ $ rpeft export gguf \
377
+ --model ruby-model \
378
+ --output ruby-model.gguf \
379
+ --no-chat-template \
380
+ --llama-cpp ~/src/llama.cpp
381
+ ```
382
+
314
383
  `rpeft doctor` and `rpeft version` are also available.
315
384
 
316
385
  ## Examples
@@ -28,6 +28,7 @@ module RPEFT
28
28
  else
29
29
  line 'Backend', "NG #{backend['error']}"
30
30
  end
31
+ render_gguf(report['gguf'])
31
32
 
32
33
  # Uppercase for consistency with the OK/MISSING library markers;
33
34
  # the check API itself keeps the machine-readable lowercase value.
@@ -35,9 +36,11 @@ module RPEFT
35
36
  render_hint(report)
36
37
  end
37
38
 
39
+ # Values come from the environment, such as LLAMA_CPP_DIR and the directories on PATH, which may hold
40
+ # escape sequences a terminal would act on.
38
41
  def line(label, value)
39
42
  padded = "#{label}:".ljust(14)
40
- @out.puts "#{padded}#{value}"
43
+ @out.puts "#{padded}#{printable(value)}"
41
44
  end
42
45
 
43
46
  def render_python(python)
@@ -65,6 +68,20 @@ module RPEFT
65
68
  line 'Device', "#{device['recommended']} (CUDA: #{cuda}, MPS: #{mps})"
66
69
  end
67
70
 
71
+ def render_gguf(gguf)
72
+ gguf ||= {}
73
+
74
+ if gguf['converter']
75
+ line 'llama.cpp', "OK #{gguf['llama_cpp_dir']}"
76
+ elsif gguf['llama_cpp_dir']
77
+ line 'llama.cpp', "NG no #{GGUF::CONVERTER} in #{gguf['llama_cpp_dir']} (LLAMA_CPP_DIR)"
78
+ else
79
+ line 'llama.cpp', 'not configured (optional, set LLAMA_CPP_DIR for GGUF export)'
80
+ end
81
+
82
+ line 'Quantizer', gguf['quantizer'] ? "OK #{gguf['quantizer']}" : "no #{GGUF::QUANTIZER} (optional, for q6_k, q5_k_m, and q4_k_m)"
83
+ end
84
+
68
85
  def render_hint(report)
69
86
  return unless report['status'] == 'degraded'
70
87
 
@@ -0,0 +1,51 @@
1
+ # frozen_string_literal: true
2
+
3
+ module RPEFT
4
+ module CLI
5
+ # @api private
6
+ class ExportGGUF < Command
7
+ REQUIRED_OPTIONS = %i(model output).freeze
8
+
9
+ private
10
+
11
+ def option_parser(options)
12
+ OptionParser.new do |o|
13
+ o.banner = 'Usage: rpeft export gguf --model DIR --output FILE [options]'
14
+ o.on('--model DIR', 'Model directory to export, such as a merged model') do |v|
15
+ options[:model] = v
16
+ end
17
+ o.on('--output FILE', 'GGUF file to write') do |v|
18
+ options[:output] = v
19
+ end
20
+ o.on('--quantize TYPE', "#{GGUF::TYPES.join(', ')}, in any case (default: q8_0)") do |v|
21
+ type = v.downcase.to_sym
22
+ raise OptionParser::InvalidArgument, v unless GGUF::TYPES.include?(type)
23
+
24
+ options[:quantize] = type
25
+ end
26
+ o.on('--no-chat-template', 'Leave out the chat template, for a model fine-tuned on plain text') do
27
+ options[:chat_template] = false
28
+ end
29
+ o.on('--llama-cpp DIR', 'llama.cpp checkout holding convert_hf_to_gguf.py (default: $LLAMA_CPP_DIR)') do |v|
30
+ options[:llama_cpp] = v
31
+ end
32
+ o.on('--llama-quantize PATH', 'llama-quantize executable for q6_k, q5_k_m, and q4_k_m (default: from PATH)') do |v|
33
+ options[:llama_quantize] = v
34
+ end
35
+ end
36
+ end
37
+
38
+ def required_options
39
+ REQUIRED_OPTIONS
40
+ end
41
+
42
+ def execute(options)
43
+ export_options = options.slice(:quantize, :chat_template, :llama_cpp, :llama_quantize)
44
+ path = Model.new(path: options[:model]).export_gguf(options[:output], **export_options)
45
+
46
+ @out.puts "GGUF saved to #{printable(path)}"
47
+ 0
48
+ end
49
+ end
50
+ end
51
+ end
data/lib/rpeft/cli.rb CHANGED
@@ -6,6 +6,7 @@ require_relative 'cli/lora_train'
6
6
  require_relative 'cli/lora_merge'
7
7
  require_relative 'cli/generate'
8
8
  require_relative 'cli/doctor'
9
+ require_relative 'cli/export_gguf'
9
10
 
10
11
  module RPEFT
11
12
  # Command line interface. Commands go through the public Ruby API only,
@@ -18,6 +19,7 @@ module RPEFT
18
19
  lora train Fine-tune a base model with LoRA
19
20
  lora merge Merge a LoRA adapter into its base model
20
21
  generate Generate text from a base or fine-tuned model
22
+ export gguf Export a model to a GGUF file for llama.cpp and Ollama
21
23
  doctor Diagnose the Python environment
22
24
  version Print the RPEFT version
23
25
  help Show this message
@@ -33,11 +35,19 @@ module RPEFT
33
35
  merge Merge a LoRA adapter into its base model
34
36
  USAGE
35
37
 
38
+ EXPORT_USAGE = <<~USAGE
39
+ Usage: rpeft export <format> [options]
40
+
41
+ Formats:
42
+ gguf Export a model to a GGUF file for llama.cpp and Ollama
43
+ USAGE
44
+
36
45
  class << self
37
46
  def run(argv, out: $stdout, err: $stderr)
38
47
  command, *rest = argv
39
48
  case command
40
49
  when 'lora' then run_lora(rest, out: out, err: err)
50
+ when 'export' then run_export(rest, out: out, err: err)
41
51
  when 'generate' then Generate.run(rest, out: out, err: err)
42
52
  when 'doctor' then Doctor.run(rest, out: out, err: err)
43
53
  when 'version', '--version', '-v'
@@ -72,6 +82,21 @@ module RPEFT
72
82
  2
73
83
  end
74
84
  end
85
+
86
+ def run_export(argv, out:, err:)
87
+ format, *rest = argv
88
+
89
+ case format
90
+ when 'gguf' then ExportGGUF.run(rest, out: out, err: err)
91
+ when nil, 'help', '--help', '-h'
92
+ out.puts EXPORT_USAGE
93
+ 0
94
+ else
95
+ err.puts "rpeft: unknown export format #{format.inspect}"
96
+ err.puts EXPORT_USAGE
97
+ 2
98
+ end
99
+ end
75
100
  end
76
101
  end
77
102
  end
@@ -12,7 +12,8 @@ module RPEFT
12
12
  end
13
13
 
14
14
  def report
15
- base = {'ruby' => {'version' => RUBY_VERSION, 'rpeft' => VERSION}}
15
+ # The llama.cpp tools are optional, for GGUF export only, so they never affect the status.
16
+ base = {'ruby' => {'version' => RUBY_VERSION, 'rpeft' => VERSION}, 'gguf' => GGUF.tools}
16
17
  begin
17
18
  diagnostics = @backend.diagnostics
18
19
  rescue Error => e
data/lib/rpeft/errors.rb CHANGED
@@ -23,4 +23,7 @@ module RPEFT
23
23
 
24
24
  # Text generation failed.
25
25
  class GenerationError < Error; end
26
+
27
+ # Exporting a model to another format, such as GGUF, failed in the external tool doing the conversion.
28
+ class ExportError < Error; end
26
29
  end
data/lib/rpeft/gguf.rb ADDED
@@ -0,0 +1,353 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'find'
4
+ require 'json'
5
+ require 'open3'
6
+ require 'tmpdir'
7
+
8
+ module RPEFT
9
+ # Exports a merged model directory to a single GGUF file for llama.cpp, Ollama, and other local runtimes.
10
+ #
11
+ # The conversion is left to llama.cpp's own tools, which know every architecture and tokenizer GGUF supports:
12
+ # its `convert_hf_to_gguf.py` writes 16-bit or Q8_0 files, and `llama-quantize` turns a 16-bit file into
13
+ # the K-quant types. RPEFT checks the input, runs the tools, and cleans up after them.
14
+ module GGUF
15
+ CONVERTER = 'convert_hf_to_gguf.py'
16
+ QUANTIZER = 'llama-quantize'
17
+ METADATA_EDITOR = File.join('gguf-py', 'gguf', 'scripts', 'gguf_new_metadata.py')
18
+ CHAT_TEMPLATE_KEY = 'tokenizer.chat_template'
19
+ # The list of a model's named templates, such as one for tool use. The templates themselves sit under keys
20
+ # holding their names, which only a reader of the file would know, so they stay; the default one is what Ollama was seen to use.
21
+ CHAT_TEMPLATE_NAMES_KEY = 'tokenizer.chat_templates'
22
+ MAGIC = 'GGUF'
23
+
24
+ # Types the converter writes directly, by its --outtype name.
25
+ CONVERTED = {f16: 'f16', bf16: 'bf16', q8_0: 'q8_0'}.freeze
26
+ # Types that need a 16-bit file run through llama-quantize, by its type name.
27
+ QUANTIZED = {q6_k: 'Q6_K', q5_k_m: 'Q5_K_M', q4_k_m: 'Q4_K_M'}.freeze
28
+ TYPES = [*CONVERTED.keys, *QUANTIZED.keys].freeze
29
+ MAX_JSON_BYTES = 256 * 1024 * 1024
30
+
31
+ # Runs a command given as an argument list, never through a shell, with extra environment variables,
32
+ # and returns its combined output and whether it succeeded. Replaced in tests so that no external tool runs.
33
+ DEFAULT_RUNNER = lambda do |argv, env|
34
+ output, status = Open3.capture2e(env, *argv)
35
+
36
+ [output, status.success?]
37
+ end
38
+
39
+ class << self
40
+ attr_writer :runner
41
+
42
+ def runner
43
+ @runner || DEFAULT_RUNNER
44
+ end
45
+
46
+ def export(model_dir, output, quantize: :q8_0, chat_template: true, llama_cpp: nil, llama_quantize: nil)
47
+ type = quantize_type(quantize)
48
+ raise ConfigurationError, "chat_template must be true or false, got #{chat_template.inspect}" unless [true, false].include?(chat_template)
49
+
50
+ model_dir = model_directory(model_dir)
51
+ output = output_path(output)
52
+ llama_cpp_dir = llama_cpp_directory(llama_cpp)
53
+ converter = converter_path(llama_cpp_dir)
54
+ quantizer = quantizer_path(llama_quantize, llama_cpp_dir) if QUANTIZED.key?(type)
55
+ editor = metadata_editor_path(llama_cpp_dir) unless chat_template
56
+
57
+ # The tools write into a directory of their own next to the output, under fixed names: the converter
58
+ # expands placeholders such as {ftype} in the name it is given, so it never sees the caller's name.
59
+ # Only a verified file reaches the output, and everything else goes with the directory, including the bf16
60
+ # intermediate of the K-quant types, which is as large as the model and so belongs on the output's disk.
61
+ Dir.mktmpdir('.rpeft-gguf-', File.dirname(output)) do |dir|
62
+ staged = File.join(dir, 'model.gguf')
63
+ env = converter_env(dir)
64
+
65
+ if quantizer
66
+ intermediate = File.join(dir, 'model-bf16.gguf')
67
+ convert(model_dir, intermediate, 'bf16', converter, env)
68
+ run([quantizer, intermediate, staged, QUANTIZED.fetch(type)], QUANTIZER)
69
+ else
70
+ convert(model_dir, staged, CONVERTED.fetch(type), converter, env)
71
+ end
72
+
73
+ verify(staged, quantizer ? QUANTIZER : CONVERTER)
74
+ staged = remove_chat_template(staged, editor, env) if editor
75
+
76
+ publish(staged, output)
77
+ end
78
+
79
+ output
80
+ rescue SystemCallError => e
81
+ raise ExportError, "GGUF export to #{output} failed: #{e.message}"
82
+ end
83
+
84
+ # The llama.cpp tools as RPEFT would find them now, for `rpeft doctor`. Never raises.
85
+ def tools(env: ENV)
86
+ configured = env.fetch('LLAMA_CPP_DIR', '').to_s
87
+ dir = configured.empty? ? nil : expand(configured) || configured
88
+ converter = dir && File.join(dir, CONVERTER)
89
+
90
+ {
91
+ 'llama_cpp_dir' => dir,
92
+ 'converter' => converter && File.file?(converter) ? converter : nil,
93
+ 'quantizer' => find_quantizer(dir, env)
94
+ }
95
+ end
96
+
97
+ private
98
+
99
+ def quantize_type(value)
100
+ type = value.to_s.downcase.to_sym if value.is_a?(String) || value.is_a?(Symbol)
101
+ return type if TYPES.include?(type)
102
+
103
+ raise ConfigurationError, "quantize must be one of #{TYPES.join(', ')}, got #{value.inspect}"
104
+ end
105
+
106
+ def model_directory(value)
107
+ dir = expanded_path(value, 'model')
108
+
109
+ unless File.directory?(dir)
110
+ raise ConfigurationError, "GGUF export needs a local model directory, got #{dir.inspect}; save a Hub model or merge an adapter into a directory first"
111
+ end
112
+
113
+ if File.file?(File.join(dir, 'adapter_config.json'))
114
+ raise ConfigurationError, "#{dir} is a LoRA adapter; merge it into its base model before exporting to GGUF"
115
+ end
116
+ raise ConfigurationError, "#{dir} has no config.json, so it is not a Transformers model directory" unless File.file?(File.join(dir, 'config.json'))
117
+
118
+ refuse_custom_code(dir)
119
+
120
+ dir
121
+ end
122
+
123
+ # A model RPEFT merged ships no custom code, so a directory that does is refused rather than trusted:
124
+ # Python files, and auto_map in any JSON file, since a config can name further config files
125
+ # (configuration_files) that Transformers reads instead. Extensions are compared in any case, as a file
126
+ # system such as macOS's default one finds configuration_x.PY when Transformers asks for
127
+ # configuration_x.py. Find covers hidden directories too and does not follow links.
128
+ #
129
+ # The check runs once, and the converter reads the directory again later, so a directory someone else can
130
+ # write to in between is not covered. Checking a private copy instead would cost a full copy of the model,
131
+ # for a directory that is normally a model the user merged; the README asks for such a directory instead.
132
+ def refuse_custom_code(dir)
133
+ # The model directory itself may be reached through a link, which Find would not enter.
134
+ root = File.realpath(dir)
135
+
136
+ Find.find(root) do |path|
137
+ next if path == root
138
+
139
+ name = path.delete_prefix("#{root}/")
140
+ refuse_entry(dir, root, name, path)
141
+ extension = File.extname(path).downcase
142
+ raise ConfigurationError, "#{dir} contains Python code (#{name}), which the GGUF converter could run" if extension == '.py'
143
+
144
+ refuse_auto_map(dir, name, path) if extension == '.json' && File.file?(path)
145
+ end
146
+ end
147
+
148
+ # The converter reads whatever a name in the directory leads to and writes some of it, such as the chat
149
+ # template, into the GGUF file. A link leading out of the directory, or a hard link, could thus carry
150
+ # another file, such as a credential, into a GGUF file that is then published, and a special file such
151
+ # as a FIFO could stall the converter. A link to a directory is refused because Find does not enter it.
152
+ def refuse_entry(dir, root, name, path)
153
+ stat = File.lstat(path)
154
+ if stat.symlink?
155
+ refuse_link(dir, root, name, path)
156
+ elsif stat.file?
157
+ raise ConfigurationError, "#{dir} holds #{name}, a hard link that may share another file's contents" if stat.nlink > 1
158
+ elsif !stat.directory?
159
+ raise ConfigurationError, "#{dir} holds #{name}, which is not a regular file"
160
+ end
161
+ end
162
+
163
+ def refuse_link(dir, root, name, path)
164
+ target = File.realpath(path)
165
+
166
+ raise ConfigurationError, "#{dir} holds #{name}, a link leading outside the directory" unless target.start_with?("#{root}/")
167
+ raise ConfigurationError, "#{dir} holds #{name}, a link to a directory, which RPEFT cannot check" if File.directory?(target)
168
+ raise ConfigurationError, "#{dir} holds #{name}, a link to something other than a regular file" unless File.file?(target)
169
+ rescue Errno::ENOENT, Errno::ELOOP
170
+ raise ConfigurationError, "#{dir} holds #{name}, a link that leads nowhere"
171
+ end
172
+
173
+ # Every JSON file is parsed rather than searched for the text auto_map, which a key can spell with
174
+ # a Unicode escape for any of its letters; a multi-megabyte tokenizer.json takes tens of milliseconds.
175
+ # NaN is allowed, as Python's json, which reads the configs, allows it.
176
+ def refuse_auto_map(dir, name, path)
177
+ # Configs and tokenizers run to tens of megabytes at most; a larger file would only be read into memory.
178
+ if File.size(path) > MAX_JSON_BYTES
179
+ raise ConfigurationError, "#{dir} holds #{name}, a JSON file larger than #{MAX_JSON_BYTES / 1024 / 1024} MB"
180
+ end
181
+
182
+ config = JSON.parse(File.read(path), allow_nan: true)
183
+ return unless config.is_a?(Hash) && config.key?('auto_map')
184
+
185
+ raise ConfigurationError, "#{dir} maps classes to custom code (auto_map in #{name}), which the GGUF converter could run"
186
+ rescue JSON::ParserError => e
187
+ raise ConfigurationError, "#{dir} holds #{name}, which is not valid JSON: #{e.message}"
188
+ end
189
+
190
+ def output_path(value)
191
+ path = expanded_path(value, 'output')
192
+
193
+ # File.exist? is false for a link whose target is missing, which the converter would write through.
194
+ if File.exist?(path) || File.symlink?(path)
195
+ raise ConfigurationError, "output #{path} already exists; remove it or choose another path"
196
+ end
197
+ raise ConfigurationError, "output directory #{File.dirname(path)} does not exist" unless File.directory?(File.dirname(path))
198
+
199
+ path
200
+ end
201
+
202
+ def llama_cpp_directory(llama_cpp)
203
+ dir = llama_cpp ? Arguments.path(llama_cpp, 'llama_cpp') : ENV.fetch('LLAMA_CPP_DIR', '')
204
+
205
+ if dir.empty?
206
+ raise DependencyError, 'GGUF export needs a llama.cpp checkout for its converter: clone ' \
207
+ 'https://github.com/ggml-org/llama.cpp and pass llama_cpp: (--llama-cpp) or set LLAMA_CPP_DIR'
208
+ end
209
+
210
+ expand(dir) || raise(ConfigurationError, "llama_cpp #{dir.inspect} cannot be expanded to a path")
211
+ end
212
+
213
+ def converter_path(llama_cpp_dir)
214
+ converter = File.join(llama_cpp_dir, CONVERTER)
215
+ raise DependencyError, "#{CONVERTER} not found in #{llama_cpp_dir}; is it a llama.cpp checkout?" unless File.file?(converter)
216
+
217
+ converter
218
+ end
219
+
220
+ def metadata_editor_path(llama_cpp_dir)
221
+ editor = File.join(llama_cpp_dir, METADATA_EDITOR)
222
+ raise DependencyError, "#{METADATA_EDITOR} not found in #{llama_cpp_dir}, needed for chat_template: false" unless File.file?(editor)
223
+
224
+ editor
225
+ end
226
+
227
+ def quantizer_path(llama_quantize, llama_cpp_dir)
228
+ if llama_quantize
229
+ path = expanded_path(llama_quantize, 'llama_quantize')
230
+ return path if File.file?(path) && File.executable?(path)
231
+
232
+ raise DependencyError, "#{path} is not an executable #{QUANTIZER}"
233
+ end
234
+
235
+ found = find_quantizer(llama_cpp_dir, ENV)
236
+ return found if found
237
+
238
+ raise DependencyError, "#{QUANTIZER} not found for the K-quant types; install llama.cpp (e.g. `brew install llama.cpp`), " \
239
+ 'build it in the llama.cpp checkout, or pass llama_quantize: (--llama-quantize)'
240
+ end
241
+
242
+ def find_quantizer(llama_cpp_dir, env)
243
+ candidates = env.fetch('PATH', '').split(File::PATH_SEPARATOR).reject(&:empty?).map { |dir| File.join(dir, QUANTIZER) }
244
+ candidates << File.join(llama_cpp_dir, 'build', 'bin', QUANTIZER) if llama_cpp_dir
245
+
246
+ candidates.find { |path| File.file?(path) && File.executable?(path) }
247
+ end
248
+
249
+ def expanded_path(value, name)
250
+ path = Arguments.path(value, name)
251
+
252
+ expand(path) || raise(ConfigurationError, "#{name} #{path.inspect} cannot be expanded to a path")
253
+ end
254
+
255
+ # File.expand_path raises for a ~user that does not exist; nil stands for that here.
256
+ def expand(path)
257
+ File.expand_path(path)
258
+ rescue ArgumentError
259
+ nil
260
+ end
261
+
262
+ def convert(model_dir, output, outtype, converter, env)
263
+ run([python, converter, model_dir, '--outfile', output, '--outtype', outtype], CONVERTER, env)
264
+ end
265
+
266
+ # Ollama (checked with 0.33.1) wraps every prompt in the chat template a GGUF file carries, even when
267
+ # a Modelfile gives a TEMPLATE of its own. A model fine-tuned on plain text expects its prompt as it was in
268
+ # training, so for such a model the template is left out. The editor writes a whole new file, skips keys
269
+ # the file does not have, and with --force skips its confirmation prompt.
270
+ def remove_chat_template(staged, editor, env)
271
+ edited = File.join(File.dirname(staged), 'model-no-chat-template.gguf')
272
+ argv = [
273
+ python, editor, '--remove-metadata', CHAT_TEMPLATE_KEY, '--remove-metadata', CHAT_TEMPLATE_NAMES_KEY, '--force', staged, edited
274
+ ]
275
+
276
+ run(argv, File.basename(editor), env)
277
+ verify(edited, File.basename(editor))
278
+
279
+ edited
280
+ end
281
+
282
+ # The converter loads some tokenizers with trust_remote_code=True. Besides the model directory, checked
283
+ # by refuse_custom_code, that code could come from the Hugging Face Hub, which offline mode keeps out,
284
+ # or from modules earlier runs left in the Hugging Face caches, which offline mode still loads. The caches
285
+ # are therefore pointed at empty directories of this export; a local model needs nothing from them.
286
+ def converter_env(dir)
287
+ {
288
+ 'HF_HUB_OFFLINE' => '1',
289
+ 'TRANSFORMERS_OFFLINE' => '1',
290
+ 'HF_HOME' => File.join(dir, 'hf-home'),
291
+ 'HF_HUB_CACHE' => File.join(dir, 'hf-hub'),
292
+ 'HF_MODULES_CACHE' => File.join(dir, 'hf-modules'),
293
+ 'TRANSFORMERS_CACHE' => nil,
294
+ 'HUGGINGFACE_HUB_CACHE' => nil
295
+ }
296
+ end
297
+
298
+ def run(argv, name, env = {})
299
+ begin
300
+ output, success = runner.call(argv, env)
301
+ rescue SystemCallError => e
302
+ raise DependencyError, "could not run #{name} (#{argv.first}): #{e.message}"
303
+ end
304
+ return if success
305
+
306
+ tail = output.to_s.lines.last(20).join
307
+
308
+ raise ExportError, "#{name} failed:\n#{tail}"
309
+ end
310
+
311
+ # The converter runs in its own process with the interpreter RPEFT uses, so it sees the same packages
312
+ # without loading torch a second time into this process through PyCall.
313
+ def python
314
+ require_relative 'backend/python'
315
+
316
+ Backend::Python.default_python || 'python3'
317
+ end
318
+
319
+ def verify(path, name)
320
+ header = File.file?(path) ? File.binread(path, MAGIC.bytesize) : nil
321
+
322
+ raise ExportError, "#{name} did not write a GGUF file" unless header == MAGIC
323
+ end
324
+
325
+ # A hard link puts the file at the output in one step and fails if anything is there by now, so a file
326
+ # another process created since output_path checked is neither replaced nor removed.
327
+ def publish(staged, output)
328
+ File.link(staged, output)
329
+ rescue Errno::EEXIST
330
+ raise ConfigurationError, "output #{output} already exists; remove it or choose another path"
331
+ rescue Errno::EPERM, Errno::ENOTSUP, Errno::EOPNOTSUPP
332
+ copy_exclusively(staged, output)
333
+ end
334
+
335
+ # Some file systems, such as exFAT, have no hard links. Creating the output exclusively keeps the same
336
+ # guarantee at the cost of a copy, and a file this creates is the only one a failed copy removes.
337
+ def copy_exclusively(staged, output)
338
+ created = false
339
+
340
+ File.open(output, File::WRONLY | File::CREAT | File::EXCL | File::BINARY) do |file|
341
+ created = true
342
+
343
+ IO.copy_stream(staged, file)
344
+ end
345
+ rescue Errno::EEXIST
346
+ raise ConfigurationError, "output #{output} already exists; remove it or choose another path"
347
+ rescue StandardError
348
+ File.delete(output) if created
349
+ raise
350
+ end
351
+ end
352
+ end
353
+ end
data/lib/rpeft/model.rb CHANGED
@@ -38,6 +38,15 @@ module RPEFT
38
38
  self
39
39
  end
40
40
 
41
+ # Writes the model to a single GGUF file for llama.cpp, Ollama, and other local runtimes, and returns its path.
42
+ # chat_template: false leaves out the chat template, for a model fine-tuned on plain text.
43
+ # See GGUF for the quantize: types and how the llama.cpp tools are found.
44
+ def export_gguf(output, quantize: :q8_0, chat_template: true, llama_cpp: nil, llama_quantize: nil)
45
+ GGUF.export(
46
+ path, output, quantize: quantize, chat_template: chat_template, llama_cpp: llama_cpp, llama_quantize: llama_quantize
47
+ )
48
+ end
49
+
41
50
  # Generates a completion for the prompt. Decoding is greedy (and thus deterministic for a given model and prompt)
42
51
  # unless temperature or top_p turns sampling on.
43
52
  def generate(prompt, max_new_tokens: 64, device: :auto, temperature: nil, top_p: nil, seed: nil)
data/lib/rpeft/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module RPEFT
4
- VERSION = '0.2.0'
4
+ VERSION = '0.3.0'
5
5
  end
data/lib/rpeft.rb CHANGED
@@ -9,6 +9,7 @@ require_relative 'rpeft/dataset'
9
9
  require_relative 'rpeft/messages'
10
10
  require_relative 'rpeft/generation'
11
11
  require_relative 'rpeft/model'
12
+ require_relative 'rpeft/gguf'
12
13
  require_relative 'rpeft/lora'
13
14
  require_relative 'rpeft/lora/adapter'
14
15
  require_relative 'rpeft/backend'
@@ -14,7 +14,7 @@ import math
14
14
  import os
15
15
  import sys
16
16
 
17
- __version__ = "0.2.0"
17
+ __version__ = "0.3.0"
18
18
 
19
19
 
20
20
  def _dumps(value):
@@ -11,8 +11,7 @@ def render(tokenizer, messages, add_generation_prompt=False):
11
11
  messages, tokenize=False, add_generation_prompt=add_generation_prompt
12
12
  )
13
13
 
14
- # Tokenizers without a chat template (small test models, some base models) still need
15
- # a deterministic rendering.
14
+ # Tokenizers without a chat template (some base models) still need a deterministic rendering.
16
15
  rendered = "\n".join(
17
16
  "%s: %s" % (message["role"], message["content"]) for message in messages
18
17
  )
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: rpeft
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Koichi ITO
@@ -60,6 +60,7 @@ files:
60
60
  - lib/rpeft/cli.rb
61
61
  - lib/rpeft/cli/command.rb
62
62
  - lib/rpeft/cli/doctor.rb
63
+ - lib/rpeft/cli/export_gguf.rb
63
64
  - lib/rpeft/cli/generate.rb
64
65
  - lib/rpeft/cli/lora_merge.rb
65
66
  - lib/rpeft/cli/lora_train.rb
@@ -69,6 +70,7 @@ files:
69
70
  - lib/rpeft/directory.rb
70
71
  - lib/rpeft/errors.rb
71
72
  - lib/rpeft/generation.rb
73
+ - lib/rpeft/gguf.rb
72
74
  - lib/rpeft/lora.rb
73
75
  - lib/rpeft/lora/adapter.rb
74
76
  - lib/rpeft/messages.rb
@@ -107,7 +109,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
107
109
  - !ruby/object:Gem::Version
108
110
  version: '0'
109
111
  requirements: []
110
- rubygems_version: 4.0.16
112
+ rubygems_version: 4.0.20
111
113
  specification_version: 4
112
114
  summary: LoRA fine-tuning for LLMs from Ruby.
113
115
  test_files: []