rpeft 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +71 -2
- data/lib/rpeft/cli/doctor.rb +18 -1
- data/lib/rpeft/cli/export_gguf.rb +51 -0
- data/lib/rpeft/cli.rb +25 -0
- data/lib/rpeft/diagnosis.rb +2 -1
- data/lib/rpeft/errors.rb +3 -0
- data/lib/rpeft/gguf.rb +353 -0
- data/lib/rpeft/model.rb +9 -0
- data/lib/rpeft/version.rb +1 -1
- data/lib/rpeft.rb +1 -0
- data/python/rpeft_backend/__init__.py +1 -1
- data/python/rpeft_backend/_chat.py +1 -2
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e72e65f59460cc4217035e8043cf25d6e2b1514638fd20589cdcf6949a9508ab
|
|
4
|
+
data.tar.gz: aeacf67c2210d3dc50508e21741badf255015cdcabcde184e8c51d1f83433b6f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 400c1b46c76250c1073488c1856967e6581b6cab2378e305208ca246ce312b4eac1066f2d9b1d1d8a2bc64bb1855f63650a516ea7af5f0aa102836d51e9545dc
|
|
7
|
+
data.tar.gz: bf4a220e57f61852471ad93319576fdab39ed06936455b5c20565f7048dd0becd3b88f49d9567957ec1566a3552b50313acc4fa000180292c2ea3cfc25a67356
|
data/README.md
CHANGED
|
@@ -12,6 +12,7 @@ It is an independent project, not affiliated with Hugging Face.
|
|
|
12
12
|
- LoRA fine-tuning from JSONL files or Ruby Enumerables, in plain text, instruction, and chat formats
|
|
13
13
|
- Merging an adapter into its base model to produce a standalone model
|
|
14
14
|
- Generation and chat with a merged model, a Hugging Face model id, or an adapter on top of its base model
|
|
15
|
+
- GGUF export of a merged model for llama.cpp, Ollama, and other local runtimes
|
|
15
16
|
- Apple Silicon (MPS) as a first-class target, with CUDA and CPU auto-detected
|
|
16
17
|
- Validation loss, per-epoch checkpoints, and resuming
|
|
17
18
|
- A CLI for training, merging, and generation, and `rpeft doctor` to check the Python environment
|
|
@@ -248,6 +249,59 @@ Apple Silicon Macs (M1 and later) are a first-class target: small models such as
|
|
|
248
249
|
Training uses float32 on MPS (half precision there is numerically fragile), bfloat16 on capable CUDA GPUs, and `PYTORCH_ENABLE_MPS_FALLBACK=1` is set automatically for
|
|
249
250
|
operators MPS does not implement. Merging always runs on CPU, which is the memory-safest place for it.
|
|
250
251
|
|
|
252
|
+
### Exporting to GGUF
|
|
253
|
+
|
|
254
|
+
A merged model can be written to a single GGUF file for llama.cpp, Ollama, and other local runtimes.
|
|
255
|
+
The conversion is done by llama.cpp's own `convert_hf_to_gguf.py`, which is not published as a package, so RPEFT needs a llama.cpp checkout:
|
|
256
|
+
|
|
257
|
+
```console
|
|
258
|
+
$ git clone https://github.com/ggml-org/llama.cpp ~/src/llama.cpp
|
|
259
|
+
$ .venv/bin/pip install sentencepiece # or, with uv: uv pip install sentencepiece
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
```ruby
|
|
263
|
+
model = adapter.merge(output_dir: './ruby-model')
|
|
264
|
+
model.export_gguf('./ruby-model.gguf', llama_cpp: '~/src/llama.cpp') # => "/path/to/ruby-model.gguf"
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
`llama_cpp:` defaults to the `LLAMA_CPP_DIR` environment variable. The converter runs in its own process with the Python RPEFT uses,
|
|
268
|
+
so the packages it needs beyond RPEFT's own go into the same environment. `sentencepiece` is one of them even for models without
|
|
269
|
+
a SentencePiece tokenizer, such as Qwen2.5, because the converter imports it before looking for one.
|
|
270
|
+
|
|
271
|
+
`quantize:` picks the type of the weights. `:q8_0` (the default), `:f16`, and `:bf16` are written by the converter directly.
|
|
272
|
+
`:q6_k`, `:q5_k_m`, and `:q4_k_m` are smaller and also need `llama-quantize`, found on `PATH` (`brew install llama.cpp`), in the checkout's `build/bin`,
|
|
273
|
+
or given as `llama_quantize:`. Small models lose more to quantization, which is why the default stays at 8 bits.
|
|
274
|
+
|
|
275
|
+
Export a model you merged yourself, from a directory no one else writes to while the export runs; an adapter is merged first.
|
|
276
|
+
An existing output file is never overwritten, and a failed export leaves nothing behind.
|
|
277
|
+
|
|
278
|
+
A GGUF file carries the model's chat template, and Ollama (checked with 0.33.1) wraps every prompt in it, even when a Modelfile gives a `TEMPLATE` of its own.
|
|
279
|
+
A model fine-tuned on plain text, like the example above, expects the prompt as it was in training, so export it without the chat template:
|
|
280
|
+
|
|
281
|
+
```ruby
|
|
282
|
+
model.export_gguf('./ruby-model.gguf', chat_template: false, llama_cpp: '~/src/llama.cpp')
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
This removes the default chat template, the one Ollama used in that check, and the list of named ones; templates stored under their own names, such as one for tool use, stay.
|
|
286
|
+
Leaving it out uses `gguf_new_metadata.py` from the same checkout and briefly needs room for one more copy of the file.
|
|
287
|
+
Keep the chat template for a model fine-tuned on chat-format data: Ollama then needs a Modelfile with the `FROM` line alone, and its chat answers as `#chat` does.
|
|
288
|
+
|
|
289
|
+
To run a file exported without its chat template in Ollama, point a Modelfile at it and give a template that passes the prompt through:
|
|
290
|
+
|
|
291
|
+
```text
|
|
292
|
+
FROM ./ruby-model.gguf
|
|
293
|
+
TEMPLATE {{ .Prompt }}
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
```console
|
|
297
|
+
$ ollama create ruby-model -f Modelfile
|
|
298
|
+
$ ollama run ruby-model $'User: How do I find the minimum value in an array?\nAssistant:'
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
The converter loads some tokenizers with `trust_remote_code=True`, so it runs offline with empty Hugging Face caches of its own, and a model directory is refused when it holds Python code,
|
|
302
|
+
`auto_map` in a JSON file, a link leading out of it or to a directory, a hard link, or a special file. Any of these could run code or carry another file, such as a credential, into the GGUF file.
|
|
303
|
+
The checks run before the converter starts, which is why the directory must not change during the export.
|
|
304
|
+
|
|
251
305
|
### Environment check
|
|
252
306
|
|
|
253
307
|
```ruby
|
|
@@ -256,7 +310,7 @@ RPEFT.check # => a Hash describing Python, packages, and devices
|
|
|
256
310
|
|
|
257
311
|
```console
|
|
258
312
|
$ rpeft doctor
|
|
259
|
-
Ruby: 4.0.
|
|
313
|
+
Ruby: 4.0.9 (rpeft 0.3.0)
|
|
260
314
|
Python: 3.12.13 (/path/to/.venv/bin/python)
|
|
261
315
|
torch: OK 2.13.0
|
|
262
316
|
transformers: OK 5.15.0
|
|
@@ -264,18 +318,23 @@ peft: OK 0.20.0
|
|
|
264
318
|
datasets: OK 5.0.1
|
|
265
319
|
accelerate: OK 1.14.0
|
|
266
320
|
Device: mps (CUDA: no, MPS: yes)
|
|
321
|
+
llama.cpp: OK /Users/you/src/llama.cpp
|
|
322
|
+
Quantizer: OK /opt/homebrew/bin/llama-quantize
|
|
267
323
|
Status: OK
|
|
268
324
|
```
|
|
269
325
|
|
|
326
|
+
The llama.cpp lines are optional, for GGUF export only, and never affect the status.
|
|
327
|
+
|
|
270
328
|
### Errors
|
|
271
329
|
|
|
272
330
|
Python exceptions never surface raw. They are translated into `RPEFT::DependencyError`, `RPEFT::ModelLoadError`, `RPEFT::DatasetError`, `RPEFT::TrainingError`,
|
|
273
331
|
`RPEFT::MergeError`, or `RPEFT::GenerationError`, and the original Python error (with its traceback) stays reachable via `error.cause`.
|
|
332
|
+
A failing llama.cpp tool during GGUF export raises `RPEFT::ExportError` with the end of the tool's output.
|
|
274
333
|
Invalid arguments are rejected in Ruby with `RPEFT::ConfigurationError` before the backend is called.
|
|
275
334
|
|
|
276
335
|
## CLI
|
|
277
336
|
|
|
278
|
-
The CLI covers the core of the lifecycle: training, merging,
|
|
337
|
+
The CLI covers the core of the lifecycle: training, merging, generation from a prompt, and GGUF export.
|
|
279
338
|
Validation datasets, checkpoints, and chat are available from the Ruby API only.
|
|
280
339
|
|
|
281
340
|
```console
|
|
@@ -311,6 +370,16 @@ $ rpeft generate \
|
|
|
311
370
|
--prompt $'User: How do I find the minimum value in an array?\nAssistant:'
|
|
312
371
|
```
|
|
313
372
|
|
|
373
|
+
A merged model is exported to GGUF with `rpeft export gguf`, which takes the same choices as `export_gguf`:
|
|
374
|
+
|
|
375
|
+
```console
|
|
376
|
+
$ rpeft export gguf \
|
|
377
|
+
--model ruby-model \
|
|
378
|
+
--output ruby-model.gguf \
|
|
379
|
+
--no-chat-template \
|
|
380
|
+
--llama-cpp ~/src/llama.cpp
|
|
381
|
+
```
|
|
382
|
+
|
|
314
383
|
`rpeft doctor` and `rpeft version` are also available.
|
|
315
384
|
|
|
316
385
|
## Examples
|
data/lib/rpeft/cli/doctor.rb
CHANGED
|
@@ -28,6 +28,7 @@ module RPEFT
|
|
|
28
28
|
else
|
|
29
29
|
line 'Backend', "NG #{backend['error']}"
|
|
30
30
|
end
|
|
31
|
+
render_gguf(report['gguf'])
|
|
31
32
|
|
|
32
33
|
# Uppercase for consistency with the OK/MISSING library markers;
|
|
33
34
|
# the check API itself keeps the machine-readable lowercase value.
|
|
@@ -35,9 +36,11 @@ module RPEFT
|
|
|
35
36
|
render_hint(report)
|
|
36
37
|
end
|
|
37
38
|
|
|
39
|
+
# Values come from the environment, such as LLAMA_CPP_DIR and the directories on PATH, which may hold
|
|
40
|
+
# escape sequences a terminal would act on.
|
|
38
41
|
def line(label, value)
|
|
39
42
|
padded = "#{label}:".ljust(14)
|
|
40
|
-
@out.puts "#{padded}#{value}"
|
|
43
|
+
@out.puts "#{padded}#{printable(value)}"
|
|
41
44
|
end
|
|
42
45
|
|
|
43
46
|
def render_python(python)
|
|
@@ -65,6 +68,20 @@ module RPEFT
|
|
|
65
68
|
line 'Device', "#{device['recommended']} (CUDA: #{cuda}, MPS: #{mps})"
|
|
66
69
|
end
|
|
67
70
|
|
|
71
|
+
def render_gguf(gguf)
|
|
72
|
+
gguf ||= {}
|
|
73
|
+
|
|
74
|
+
if gguf['converter']
|
|
75
|
+
line 'llama.cpp', "OK #{gguf['llama_cpp_dir']}"
|
|
76
|
+
elsif gguf['llama_cpp_dir']
|
|
77
|
+
line 'llama.cpp', "NG no #{GGUF::CONVERTER} in #{gguf['llama_cpp_dir']} (LLAMA_CPP_DIR)"
|
|
78
|
+
else
|
|
79
|
+
line 'llama.cpp', 'not configured (optional, set LLAMA_CPP_DIR for GGUF export)'
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
line 'Quantizer', gguf['quantizer'] ? "OK #{gguf['quantizer']}" : "no #{GGUF::QUANTIZER} (optional, for q6_k, q5_k_m, and q4_k_m)"
|
|
83
|
+
end
|
|
84
|
+
|
|
68
85
|
def render_hint(report)
|
|
69
86
|
return unless report['status'] == 'degraded'
|
|
70
87
|
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RPEFT
|
|
4
|
+
module CLI
|
|
5
|
+
# @api private
|
|
6
|
+
class ExportGGUF < Command
|
|
7
|
+
REQUIRED_OPTIONS = %i(model output).freeze
|
|
8
|
+
|
|
9
|
+
private
|
|
10
|
+
|
|
11
|
+
def option_parser(options)
|
|
12
|
+
OptionParser.new do |o|
|
|
13
|
+
o.banner = 'Usage: rpeft export gguf --model DIR --output FILE [options]'
|
|
14
|
+
o.on('--model DIR', 'Model directory to export, such as a merged model') do |v|
|
|
15
|
+
options[:model] = v
|
|
16
|
+
end
|
|
17
|
+
o.on('--output FILE', 'GGUF file to write') do |v|
|
|
18
|
+
options[:output] = v
|
|
19
|
+
end
|
|
20
|
+
o.on('--quantize TYPE', "#{GGUF::TYPES.join(', ')}, in any case (default: q8_0)") do |v|
|
|
21
|
+
type = v.downcase.to_sym
|
|
22
|
+
raise OptionParser::InvalidArgument, v unless GGUF::TYPES.include?(type)
|
|
23
|
+
|
|
24
|
+
options[:quantize] = type
|
|
25
|
+
end
|
|
26
|
+
o.on('--no-chat-template', 'Leave out the chat template, for a model fine-tuned on plain text') do
|
|
27
|
+
options[:chat_template] = false
|
|
28
|
+
end
|
|
29
|
+
o.on('--llama-cpp DIR', 'llama.cpp checkout holding convert_hf_to_gguf.py (default: $LLAMA_CPP_DIR)') do |v|
|
|
30
|
+
options[:llama_cpp] = v
|
|
31
|
+
end
|
|
32
|
+
o.on('--llama-quantize PATH', 'llama-quantize executable for q6_k, q5_k_m, and q4_k_m (default: from PATH)') do |v|
|
|
33
|
+
options[:llama_quantize] = v
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def required_options
|
|
39
|
+
REQUIRED_OPTIONS
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def execute(options)
|
|
43
|
+
export_options = options.slice(:quantize, :chat_template, :llama_cpp, :llama_quantize)
|
|
44
|
+
path = Model.new(path: options[:model]).export_gguf(options[:output], **export_options)
|
|
45
|
+
|
|
46
|
+
@out.puts "GGUF saved to #{printable(path)}"
|
|
47
|
+
0
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
end
|
data/lib/rpeft/cli.rb
CHANGED
|
@@ -6,6 +6,7 @@ require_relative 'cli/lora_train'
|
|
|
6
6
|
require_relative 'cli/lora_merge'
|
|
7
7
|
require_relative 'cli/generate'
|
|
8
8
|
require_relative 'cli/doctor'
|
|
9
|
+
require_relative 'cli/export_gguf'
|
|
9
10
|
|
|
10
11
|
module RPEFT
|
|
11
12
|
# Command line interface. Commands go through the public Ruby API only,
|
|
@@ -18,6 +19,7 @@ module RPEFT
|
|
|
18
19
|
lora train Fine-tune a base model with LoRA
|
|
19
20
|
lora merge Merge a LoRA adapter into its base model
|
|
20
21
|
generate Generate text from a base or fine-tuned model
|
|
22
|
+
export gguf Export a model to a GGUF file for llama.cpp and Ollama
|
|
21
23
|
doctor Diagnose the Python environment
|
|
22
24
|
version Print the RPEFT version
|
|
23
25
|
help Show this message
|
|
@@ -33,11 +35,19 @@ module RPEFT
|
|
|
33
35
|
merge Merge a LoRA adapter into its base model
|
|
34
36
|
USAGE
|
|
35
37
|
|
|
38
|
+
EXPORT_USAGE = <<~USAGE
|
|
39
|
+
Usage: rpeft export <format> [options]
|
|
40
|
+
|
|
41
|
+
Formats:
|
|
42
|
+
gguf Export a model to a GGUF file for llama.cpp and Ollama
|
|
43
|
+
USAGE
|
|
44
|
+
|
|
36
45
|
class << self
|
|
37
46
|
def run(argv, out: $stdout, err: $stderr)
|
|
38
47
|
command, *rest = argv
|
|
39
48
|
case command
|
|
40
49
|
when 'lora' then run_lora(rest, out: out, err: err)
|
|
50
|
+
when 'export' then run_export(rest, out: out, err: err)
|
|
41
51
|
when 'generate' then Generate.run(rest, out: out, err: err)
|
|
42
52
|
when 'doctor' then Doctor.run(rest, out: out, err: err)
|
|
43
53
|
when 'version', '--version', '-v'
|
|
@@ -72,6 +82,21 @@ module RPEFT
|
|
|
72
82
|
2
|
|
73
83
|
end
|
|
74
84
|
end
|
|
85
|
+
|
|
86
|
+
def run_export(argv, out:, err:)
|
|
87
|
+
format, *rest = argv
|
|
88
|
+
|
|
89
|
+
case format
|
|
90
|
+
when 'gguf' then ExportGGUF.run(rest, out: out, err: err)
|
|
91
|
+
when nil, 'help', '--help', '-h'
|
|
92
|
+
out.puts EXPORT_USAGE
|
|
93
|
+
0
|
|
94
|
+
else
|
|
95
|
+
err.puts "rpeft: unknown export format #{format.inspect}"
|
|
96
|
+
err.puts EXPORT_USAGE
|
|
97
|
+
2
|
|
98
|
+
end
|
|
99
|
+
end
|
|
75
100
|
end
|
|
76
101
|
end
|
|
77
102
|
end
|
data/lib/rpeft/diagnosis.rb
CHANGED
|
@@ -12,7 +12,8 @@ module RPEFT
|
|
|
12
12
|
end
|
|
13
13
|
|
|
14
14
|
def report
|
|
15
|
-
|
|
15
|
+
# The llama.cpp tools are optional, for GGUF export only, so they never affect the status.
|
|
16
|
+
base = {'ruby' => {'version' => RUBY_VERSION, 'rpeft' => VERSION}, 'gguf' => GGUF.tools}
|
|
16
17
|
begin
|
|
17
18
|
diagnostics = @backend.diagnostics
|
|
18
19
|
rescue Error => e
|
data/lib/rpeft/errors.rb
CHANGED
data/lib/rpeft/gguf.rb
ADDED
|
@@ -0,0 +1,353 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'find'
|
|
4
|
+
require 'json'
|
|
5
|
+
require 'open3'
|
|
6
|
+
require 'tmpdir'
|
|
7
|
+
|
|
8
|
+
module RPEFT
|
|
9
|
+
# Exports a merged model directory to a single GGUF file for llama.cpp, Ollama, and other local runtimes.
|
|
10
|
+
#
|
|
11
|
+
# The conversion is left to llama.cpp's own tools, which know every architecture and tokenizer GGUF supports:
|
|
12
|
+
# its `convert_hf_to_gguf.py` writes 16-bit or Q8_0 files, and `llama-quantize` turns a 16-bit file into
|
|
13
|
+
# the K-quant types. RPEFT checks the input, runs the tools, and cleans up after them.
|
|
14
|
+
module GGUF
|
|
15
|
+
CONVERTER = 'convert_hf_to_gguf.py'
|
|
16
|
+
QUANTIZER = 'llama-quantize'
|
|
17
|
+
METADATA_EDITOR = File.join('gguf-py', 'gguf', 'scripts', 'gguf_new_metadata.py')
|
|
18
|
+
CHAT_TEMPLATE_KEY = 'tokenizer.chat_template'
|
|
19
|
+
# The list of a model's named templates, such as one for tool use. The templates themselves sit under keys
|
|
20
|
+
# holding their names, which only a reader of the file would know, so they stay; the default one is what Ollama was seen to use.
|
|
21
|
+
CHAT_TEMPLATE_NAMES_KEY = 'tokenizer.chat_templates'
|
|
22
|
+
MAGIC = 'GGUF'
|
|
23
|
+
|
|
24
|
+
# Types the converter writes directly, by its --outtype name.
|
|
25
|
+
CONVERTED = {f16: 'f16', bf16: 'bf16', q8_0: 'q8_0'}.freeze
|
|
26
|
+
# Types that need a 16-bit file run through llama-quantize, by its type name.
|
|
27
|
+
QUANTIZED = {q6_k: 'Q6_K', q5_k_m: 'Q5_K_M', q4_k_m: 'Q4_K_M'}.freeze
|
|
28
|
+
TYPES = [*CONVERTED.keys, *QUANTIZED.keys].freeze
|
|
29
|
+
MAX_JSON_BYTES = 256 * 1024 * 1024
|
|
30
|
+
|
|
31
|
+
# Runs a command given as an argument list, never through a shell, with extra environment variables,
|
|
32
|
+
# and returns its combined output and whether it succeeded. Replaced in tests so that no external tool runs.
|
|
33
|
+
DEFAULT_RUNNER = lambda do |argv, env|
|
|
34
|
+
output, status = Open3.capture2e(env, *argv)
|
|
35
|
+
|
|
36
|
+
[output, status.success?]
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
class << self
|
|
40
|
+
attr_writer :runner
|
|
41
|
+
|
|
42
|
+
def runner
|
|
43
|
+
@runner || DEFAULT_RUNNER
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def export(model_dir, output, quantize: :q8_0, chat_template: true, llama_cpp: nil, llama_quantize: nil)
|
|
47
|
+
type = quantize_type(quantize)
|
|
48
|
+
raise ConfigurationError, "chat_template must be true or false, got #{chat_template.inspect}" unless [true, false].include?(chat_template)
|
|
49
|
+
|
|
50
|
+
model_dir = model_directory(model_dir)
|
|
51
|
+
output = output_path(output)
|
|
52
|
+
llama_cpp_dir = llama_cpp_directory(llama_cpp)
|
|
53
|
+
converter = converter_path(llama_cpp_dir)
|
|
54
|
+
quantizer = quantizer_path(llama_quantize, llama_cpp_dir) if QUANTIZED.key?(type)
|
|
55
|
+
editor = metadata_editor_path(llama_cpp_dir) unless chat_template
|
|
56
|
+
|
|
57
|
+
# The tools write into a directory of their own next to the output, under fixed names: the converter
|
|
58
|
+
# expands placeholders such as {ftype} in the name it is given, so it never sees the caller's name.
|
|
59
|
+
# Only a verified file reaches the output, and everything else goes with the directory, including the bf16
|
|
60
|
+
# intermediate of the K-quant types, which is as large as the model and so belongs on the output's disk.
|
|
61
|
+
Dir.mktmpdir('.rpeft-gguf-', File.dirname(output)) do |dir|
|
|
62
|
+
staged = File.join(dir, 'model.gguf')
|
|
63
|
+
env = converter_env(dir)
|
|
64
|
+
|
|
65
|
+
if quantizer
|
|
66
|
+
intermediate = File.join(dir, 'model-bf16.gguf')
|
|
67
|
+
convert(model_dir, intermediate, 'bf16', converter, env)
|
|
68
|
+
run([quantizer, intermediate, staged, QUANTIZED.fetch(type)], QUANTIZER)
|
|
69
|
+
else
|
|
70
|
+
convert(model_dir, staged, CONVERTED.fetch(type), converter, env)
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
verify(staged, quantizer ? QUANTIZER : CONVERTER)
|
|
74
|
+
staged = remove_chat_template(staged, editor, env) if editor
|
|
75
|
+
|
|
76
|
+
publish(staged, output)
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
output
|
|
80
|
+
rescue SystemCallError => e
|
|
81
|
+
raise ExportError, "GGUF export to #{output} failed: #{e.message}"
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
# The llama.cpp tools as RPEFT would find them now, for `rpeft doctor`. Never raises.
|
|
85
|
+
def tools(env: ENV)
|
|
86
|
+
configured = env.fetch('LLAMA_CPP_DIR', '').to_s
|
|
87
|
+
dir = configured.empty? ? nil : expand(configured) || configured
|
|
88
|
+
converter = dir && File.join(dir, CONVERTER)
|
|
89
|
+
|
|
90
|
+
{
|
|
91
|
+
'llama_cpp_dir' => dir,
|
|
92
|
+
'converter' => converter && File.file?(converter) ? converter : nil,
|
|
93
|
+
'quantizer' => find_quantizer(dir, env)
|
|
94
|
+
}
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
private
|
|
98
|
+
|
|
99
|
+
def quantize_type(value)
|
|
100
|
+
type = value.to_s.downcase.to_sym if value.is_a?(String) || value.is_a?(Symbol)
|
|
101
|
+
return type if TYPES.include?(type)
|
|
102
|
+
|
|
103
|
+
raise ConfigurationError, "quantize must be one of #{TYPES.join(', ')}, got #{value.inspect}"
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
def model_directory(value)
|
|
107
|
+
dir = expanded_path(value, 'model')
|
|
108
|
+
|
|
109
|
+
unless File.directory?(dir)
|
|
110
|
+
raise ConfigurationError, "GGUF export needs a local model directory, got #{dir.inspect}; save a Hub model or merge an adapter into a directory first"
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
if File.file?(File.join(dir, 'adapter_config.json'))
|
|
114
|
+
raise ConfigurationError, "#{dir} is a LoRA adapter; merge it into its base model before exporting to GGUF"
|
|
115
|
+
end
|
|
116
|
+
raise ConfigurationError, "#{dir} has no config.json, so it is not a Transformers model directory" unless File.file?(File.join(dir, 'config.json'))
|
|
117
|
+
|
|
118
|
+
refuse_custom_code(dir)
|
|
119
|
+
|
|
120
|
+
dir
|
|
121
|
+
end
|
|
122
|
+
|
|
123
|
+
# A model RPEFT merged ships no custom code, so a directory that does is refused rather than trusted:
|
|
124
|
+
# Python files, and auto_map in any JSON file, since a config can name further config files
|
|
125
|
+
# (configuration_files) that Transformers reads instead. Extensions are compared in any case, as a file
|
|
126
|
+
# system such as macOS's default one finds configuration_x.PY when Transformers asks for
|
|
127
|
+
# configuration_x.py. Find covers hidden directories too and does not follow links.
|
|
128
|
+
#
|
|
129
|
+
# The check runs once, and the converter reads the directory again later, so a directory someone else can
|
|
130
|
+
# write to in between is not covered. Checking a private copy instead would cost a full copy of the model,
|
|
131
|
+
# for a directory that is normally a model the user merged; the README asks for such a directory instead.
|
|
132
|
+
def refuse_custom_code(dir)
|
|
133
|
+
# The model directory itself may be reached through a link, which Find would not enter.
|
|
134
|
+
root = File.realpath(dir)
|
|
135
|
+
|
|
136
|
+
Find.find(root) do |path|
|
|
137
|
+
next if path == root
|
|
138
|
+
|
|
139
|
+
name = path.delete_prefix("#{root}/")
|
|
140
|
+
refuse_entry(dir, root, name, path)
|
|
141
|
+
extension = File.extname(path).downcase
|
|
142
|
+
raise ConfigurationError, "#{dir} contains Python code (#{name}), which the GGUF converter could run" if extension == '.py'
|
|
143
|
+
|
|
144
|
+
refuse_auto_map(dir, name, path) if extension == '.json' && File.file?(path)
|
|
145
|
+
end
|
|
146
|
+
end
|
|
147
|
+
|
|
148
|
+
# The converter reads whatever a name in the directory leads to and writes some of it, such as the chat
|
|
149
|
+
# template, into the GGUF file. A link leading out of the directory, or a hard link, could thus carry
|
|
150
|
+
# another file, such as a credential, into a GGUF file that is then published, and a special file such
|
|
151
|
+
# as a FIFO could stall the converter. A link to a directory is refused because Find does not enter it.
|
|
152
|
+
def refuse_entry(dir, root, name, path)
|
|
153
|
+
stat = File.lstat(path)
|
|
154
|
+
if stat.symlink?
|
|
155
|
+
refuse_link(dir, root, name, path)
|
|
156
|
+
elsif stat.file?
|
|
157
|
+
raise ConfigurationError, "#{dir} holds #{name}, a hard link that may share another file's contents" if stat.nlink > 1
|
|
158
|
+
elsif !stat.directory?
|
|
159
|
+
raise ConfigurationError, "#{dir} holds #{name}, which is not a regular file"
|
|
160
|
+
end
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
def refuse_link(dir, root, name, path)
|
|
164
|
+
target = File.realpath(path)
|
|
165
|
+
|
|
166
|
+
raise ConfigurationError, "#{dir} holds #{name}, a link leading outside the directory" unless target.start_with?("#{root}/")
|
|
167
|
+
raise ConfigurationError, "#{dir} holds #{name}, a link to a directory, which RPEFT cannot check" if File.directory?(target)
|
|
168
|
+
raise ConfigurationError, "#{dir} holds #{name}, a link to something other than a regular file" unless File.file?(target)
|
|
169
|
+
rescue Errno::ENOENT, Errno::ELOOP
|
|
170
|
+
raise ConfigurationError, "#{dir} holds #{name}, a link that leads nowhere"
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
# Every JSON file is parsed rather than searched for the text auto_map, which a key can spell with
|
|
174
|
+
# a Unicode escape for any of its letters; a multi-megabyte tokenizer.json takes tens of milliseconds.
|
|
175
|
+
# NaN is allowed, as Python's json, which reads the configs, allows it.
|
|
176
|
+
def refuse_auto_map(dir, name, path)
|
|
177
|
+
# Configs and tokenizers run to tens of megabytes at most; a larger file would only be read into memory.
|
|
178
|
+
if File.size(path) > MAX_JSON_BYTES
|
|
179
|
+
raise ConfigurationError, "#{dir} holds #{name}, a JSON file larger than #{MAX_JSON_BYTES / 1024 / 1024} MB"
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
config = JSON.parse(File.read(path), allow_nan: true)
|
|
183
|
+
return unless config.is_a?(Hash) && config.key?('auto_map')
|
|
184
|
+
|
|
185
|
+
raise ConfigurationError, "#{dir} maps classes to custom code (auto_map in #{name}), which the GGUF converter could run"
|
|
186
|
+
rescue JSON::ParserError => e
|
|
187
|
+
raise ConfigurationError, "#{dir} holds #{name}, which is not valid JSON: #{e.message}"
|
|
188
|
+
end
|
|
189
|
+
|
|
190
|
+
def output_path(value)
|
|
191
|
+
path = expanded_path(value, 'output')
|
|
192
|
+
|
|
193
|
+
# File.exist? is false for a link whose target is missing, which the converter would write through.
|
|
194
|
+
if File.exist?(path) || File.symlink?(path)
|
|
195
|
+
raise ConfigurationError, "output #{path} already exists; remove it or choose another path"
|
|
196
|
+
end
|
|
197
|
+
raise ConfigurationError, "output directory #{File.dirname(path)} does not exist" unless File.directory?(File.dirname(path))
|
|
198
|
+
|
|
199
|
+
path
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
def llama_cpp_directory(llama_cpp)
|
|
203
|
+
dir = llama_cpp ? Arguments.path(llama_cpp, 'llama_cpp') : ENV.fetch('LLAMA_CPP_DIR', '')
|
|
204
|
+
|
|
205
|
+
if dir.empty?
|
|
206
|
+
raise DependencyError, 'GGUF export needs a llama.cpp checkout for its converter: clone ' \
|
|
207
|
+
'https://github.com/ggml-org/llama.cpp and pass llama_cpp: (--llama-cpp) or set LLAMA_CPP_DIR'
|
|
208
|
+
end
|
|
209
|
+
|
|
210
|
+
expand(dir) || raise(ConfigurationError, "llama_cpp #{dir.inspect} cannot be expanded to a path")
|
|
211
|
+
end
|
|
212
|
+
|
|
213
|
+
def converter_path(llama_cpp_dir)
|
|
214
|
+
converter = File.join(llama_cpp_dir, CONVERTER)
|
|
215
|
+
raise DependencyError, "#{CONVERTER} not found in #{llama_cpp_dir}; is it a llama.cpp checkout?" unless File.file?(converter)
|
|
216
|
+
|
|
217
|
+
converter
|
|
218
|
+
end
|
|
219
|
+
|
|
220
|
+
def metadata_editor_path(llama_cpp_dir)
|
|
221
|
+
editor = File.join(llama_cpp_dir, METADATA_EDITOR)
|
|
222
|
+
raise DependencyError, "#{METADATA_EDITOR} not found in #{llama_cpp_dir}, needed for chat_template: false" unless File.file?(editor)
|
|
223
|
+
|
|
224
|
+
editor
|
|
225
|
+
end
|
|
226
|
+
|
|
227
|
+
def quantizer_path(llama_quantize, llama_cpp_dir)
|
|
228
|
+
if llama_quantize
|
|
229
|
+
path = expanded_path(llama_quantize, 'llama_quantize')
|
|
230
|
+
return path if File.file?(path) && File.executable?(path)
|
|
231
|
+
|
|
232
|
+
raise DependencyError, "#{path} is not an executable #{QUANTIZER}"
|
|
233
|
+
end
|
|
234
|
+
|
|
235
|
+
found = find_quantizer(llama_cpp_dir, ENV)
|
|
236
|
+
return found if found
|
|
237
|
+
|
|
238
|
+
raise DependencyError, "#{QUANTIZER} not found for the K-quant types; install llama.cpp (e.g. `brew install llama.cpp`), " \
|
|
239
|
+
'build it in the llama.cpp checkout, or pass llama_quantize: (--llama-quantize)'
|
|
240
|
+
end
|
|
241
|
+
|
|
242
|
+
def find_quantizer(llama_cpp_dir, env)
|
|
243
|
+
candidates = env.fetch('PATH', '').split(File::PATH_SEPARATOR).reject(&:empty?).map { |dir| File.join(dir, QUANTIZER) }
|
|
244
|
+
candidates << File.join(llama_cpp_dir, 'build', 'bin', QUANTIZER) if llama_cpp_dir
|
|
245
|
+
|
|
246
|
+
candidates.find { |path| File.file?(path) && File.executable?(path) }
|
|
247
|
+
end
|
|
248
|
+
|
|
249
|
+
def expanded_path(value, name)
|
|
250
|
+
path = Arguments.path(value, name)
|
|
251
|
+
|
|
252
|
+
expand(path) || raise(ConfigurationError, "#{name} #{path.inspect} cannot be expanded to a path")
|
|
253
|
+
end
|
|
254
|
+
|
|
255
|
+
# File.expand_path raises for a ~user that does not exist; nil stands for that here.
|
|
256
|
+
def expand(path)
|
|
257
|
+
File.expand_path(path)
|
|
258
|
+
rescue ArgumentError
|
|
259
|
+
nil
|
|
260
|
+
end
|
|
261
|
+
|
|
262
|
+
def convert(model_dir, output, outtype, converter, env)
|
|
263
|
+
run([python, converter, model_dir, '--outfile', output, '--outtype', outtype], CONVERTER, env)
|
|
264
|
+
end
|
|
265
|
+
|
|
266
|
+
# Ollama (checked with 0.33.1) wraps every prompt in the chat template a GGUF file carries, even when
|
|
267
|
+
# a Modelfile gives a TEMPLATE of its own. A model fine-tuned on plain text expects its prompt as it was in
|
|
268
|
+
# training, so for such a model the template is left out. The editor writes a whole new file, skips keys
|
|
269
|
+
# the file does not have, and with --force skips its confirmation prompt.
|
|
270
|
+
def remove_chat_template(staged, editor, env)
|
|
271
|
+
edited = File.join(File.dirname(staged), 'model-no-chat-template.gguf')
|
|
272
|
+
argv = [
|
|
273
|
+
python, editor, '--remove-metadata', CHAT_TEMPLATE_KEY, '--remove-metadata', CHAT_TEMPLATE_NAMES_KEY, '--force', staged, edited
|
|
274
|
+
]
|
|
275
|
+
|
|
276
|
+
run(argv, File.basename(editor), env)
|
|
277
|
+
verify(edited, File.basename(editor))
|
|
278
|
+
|
|
279
|
+
edited
|
|
280
|
+
end
|
|
281
|
+
|
|
282
|
+
# The converter loads some tokenizers with trust_remote_code=True. Besides the model directory, checked
|
|
283
|
+
# by refuse_custom_code, that code could come from the Hugging Face Hub, which offline mode keeps out,
|
|
284
|
+
# or from modules earlier runs left in the Hugging Face caches, which offline mode still loads. The caches
|
|
285
|
+
# are therefore pointed at empty directories of this export; a local model needs nothing from them.
|
|
286
|
+
def converter_env(dir)
|
|
287
|
+
{
|
|
288
|
+
'HF_HUB_OFFLINE' => '1',
|
|
289
|
+
'TRANSFORMERS_OFFLINE' => '1',
|
|
290
|
+
'HF_HOME' => File.join(dir, 'hf-home'),
|
|
291
|
+
'HF_HUB_CACHE' => File.join(dir, 'hf-hub'),
|
|
292
|
+
'HF_MODULES_CACHE' => File.join(dir, 'hf-modules'),
|
|
293
|
+
'TRANSFORMERS_CACHE' => nil,
|
|
294
|
+
'HUGGINGFACE_HUB_CACHE' => nil
|
|
295
|
+
}
|
|
296
|
+
end
|
|
297
|
+
|
|
298
|
+
def run(argv, name, env = {})
|
|
299
|
+
begin
|
|
300
|
+
output, success = runner.call(argv, env)
|
|
301
|
+
rescue SystemCallError => e
|
|
302
|
+
raise DependencyError, "could not run #{name} (#{argv.first}): #{e.message}"
|
|
303
|
+
end
|
|
304
|
+
return if success
|
|
305
|
+
|
|
306
|
+
tail = output.to_s.lines.last(20).join
|
|
307
|
+
|
|
308
|
+
raise ExportError, "#{name} failed:\n#{tail}"
|
|
309
|
+
end
|
|
310
|
+
|
|
311
|
+
# The converter runs in its own process with the interpreter RPEFT uses, so it sees the same packages
|
|
312
|
+
# without loading torch a second time into this process through PyCall.
|
|
313
|
+
def python
|
|
314
|
+
require_relative 'backend/python'
|
|
315
|
+
|
|
316
|
+
Backend::Python.default_python || 'python3'
|
|
317
|
+
end
|
|
318
|
+
|
|
319
|
+
def verify(path, name)
|
|
320
|
+
header = File.file?(path) ? File.binread(path, MAGIC.bytesize) : nil
|
|
321
|
+
|
|
322
|
+
raise ExportError, "#{name} did not write a GGUF file" unless header == MAGIC
|
|
323
|
+
end
|
|
324
|
+
|
|
325
|
+
# A hard link puts the file at the output in one step and fails if anything is there by now, so a file
|
|
326
|
+
# another process created since output_path checked is neither replaced nor removed.
|
|
327
|
+
def publish(staged, output)
|
|
328
|
+
File.link(staged, output)
|
|
329
|
+
rescue Errno::EEXIST
|
|
330
|
+
raise ConfigurationError, "output #{output} already exists; remove it or choose another path"
|
|
331
|
+
rescue Errno::EPERM, Errno::ENOTSUP, Errno::EOPNOTSUPP
|
|
332
|
+
copy_exclusively(staged, output)
|
|
333
|
+
end
|
|
334
|
+
|
|
335
|
+
# Some file systems, such as exFAT, have no hard links. Creating the output exclusively keeps the same
|
|
336
|
+
# guarantee at the cost of a copy, and a file this creates is the only one a failed copy removes.
|
|
337
|
+
def copy_exclusively(staged, output)
|
|
338
|
+
created = false
|
|
339
|
+
|
|
340
|
+
File.open(output, File::WRONLY | File::CREAT | File::EXCL | File::BINARY) do |file|
|
|
341
|
+
created = true
|
|
342
|
+
|
|
343
|
+
IO.copy_stream(staged, file)
|
|
344
|
+
end
|
|
345
|
+
rescue Errno::EEXIST
|
|
346
|
+
raise ConfigurationError, "output #{output} already exists; remove it or choose another path"
|
|
347
|
+
rescue StandardError
|
|
348
|
+
File.delete(output) if created
|
|
349
|
+
raise
|
|
350
|
+
end
|
|
351
|
+
end
|
|
352
|
+
end
|
|
353
|
+
end
|
data/lib/rpeft/model.rb
CHANGED
|
@@ -38,6 +38,15 @@ module RPEFT
|
|
|
38
38
|
self
|
|
39
39
|
end
|
|
40
40
|
|
|
41
|
+
# Writes the model to a single GGUF file for llama.cpp, Ollama, and other local runtimes, and returns its path.
|
|
42
|
+
# chat_template: false leaves out the chat template, for a model fine-tuned on plain text.
|
|
43
|
+
# See GGUF for the quantize: types and how the llama.cpp tools are found.
|
|
44
|
+
def export_gguf(output, quantize: :q8_0, chat_template: true, llama_cpp: nil, llama_quantize: nil)
|
|
45
|
+
GGUF.export(
|
|
46
|
+
path, output, quantize: quantize, chat_template: chat_template, llama_cpp: llama_cpp, llama_quantize: llama_quantize
|
|
47
|
+
)
|
|
48
|
+
end
|
|
49
|
+
|
|
41
50
|
# Generates a completion for the prompt. Decoding is greedy (and thus deterministic for a given model and prompt)
|
|
42
51
|
# unless temperature or top_p turns sampling on.
|
|
43
52
|
def generate(prompt, max_new_tokens: 64, device: :auto, temperature: nil, top_p: nil, seed: nil)
|
data/lib/rpeft/version.rb
CHANGED
data/lib/rpeft.rb
CHANGED
|
@@ -9,6 +9,7 @@ require_relative 'rpeft/dataset'
|
|
|
9
9
|
require_relative 'rpeft/messages'
|
|
10
10
|
require_relative 'rpeft/generation'
|
|
11
11
|
require_relative 'rpeft/model'
|
|
12
|
+
require_relative 'rpeft/gguf'
|
|
12
13
|
require_relative 'rpeft/lora'
|
|
13
14
|
require_relative 'rpeft/lora/adapter'
|
|
14
15
|
require_relative 'rpeft/backend'
|
|
@@ -11,8 +11,7 @@ def render(tokenizer, messages, add_generation_prompt=False):
|
|
|
11
11
|
messages, tokenize=False, add_generation_prompt=add_generation_prompt
|
|
12
12
|
)
|
|
13
13
|
|
|
14
|
-
# Tokenizers without a chat template (
|
|
15
|
-
# a deterministic rendering.
|
|
14
|
+
# Tokenizers without a chat template (some base models) still need a deterministic rendering.
|
|
16
15
|
rendered = "\n".join(
|
|
17
16
|
"%s: %s" % (message["role"], message["content"]) for message in messages
|
|
18
17
|
)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: rpeft
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Koichi ITO
|
|
@@ -60,6 +60,7 @@ files:
|
|
|
60
60
|
- lib/rpeft/cli.rb
|
|
61
61
|
- lib/rpeft/cli/command.rb
|
|
62
62
|
- lib/rpeft/cli/doctor.rb
|
|
63
|
+
- lib/rpeft/cli/export_gguf.rb
|
|
63
64
|
- lib/rpeft/cli/generate.rb
|
|
64
65
|
- lib/rpeft/cli/lora_merge.rb
|
|
65
66
|
- lib/rpeft/cli/lora_train.rb
|
|
@@ -69,6 +70,7 @@ files:
|
|
|
69
70
|
- lib/rpeft/directory.rb
|
|
70
71
|
- lib/rpeft/errors.rb
|
|
71
72
|
- lib/rpeft/generation.rb
|
|
73
|
+
- lib/rpeft/gguf.rb
|
|
72
74
|
- lib/rpeft/lora.rb
|
|
73
75
|
- lib/rpeft/lora/adapter.rb
|
|
74
76
|
- lib/rpeft/messages.rb
|
|
@@ -107,7 +109,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
107
109
|
- !ruby/object:Gem::Version
|
|
108
110
|
version: '0'
|
|
109
111
|
requirements: []
|
|
110
|
-
rubygems_version: 4.0.
|
|
112
|
+
rubygems_version: 4.0.20
|
|
111
113
|
specification_version: 4
|
|
112
114
|
summary: LoRA fine-tuning for LLMs from Ruby.
|
|
113
115
|
test_files: []
|