rllama 1.1.0-x86_64-linux-gnu → 1.2.0-x86_64-linux-gnu

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1f90defedb89874f6c356bc57e33f78122eb12ab3581161ec2a585a1de354fa1
-  data.tar.gz: 02d4fd8601f8624efb6a80f122574b40fd83706158f4656a20f85d369b79248c
+  metadata.gz: 592b171a8e0aefe8eeda202d06f0c63f79cd6e23ac32e826e31aedc53a399dfa
+  data.tar.gz: 65ea9118cd98b74cd92c1692c11354dbdd68201c594e293356535235962739a7
 SHA512:
-  metadata.gz: 53440e4b7a6f411c15cfe9d97931d3ef8278d671a68208266b28dd37203a586bb8c19d07d4144127a053a0b14bd684fc9c537d9b1d75cc42c7c04135662be1cc
-  data.tar.gz: d99fdf22266da73a99b8e299520e14ce9a78a1057da9dd158df3dc28b90609a24352ec6cb67362cf8c1d3e05a208304188cfbda62b2660784c5b41de30bb7a37
+  metadata.gz: 92a75c7a2528c42b3a1457e26b969cae6ee5e14b524d6c9c796f5265dea4e749aebc36327d0c55ef0e4d315d2c1ccb3fda353e020e2ace76ea41ed7175371a35
+  data.tar.gz: 37e02906be2644ee0fe024f78e89f1882d86cd66cf958b710b4798aae1bf99f182810fb16dd86ca950a1516a638d787cdf0f7eb256a4c44d7803bb75e6f856a6

data/LICENSE

@@ -0,0 +1,18 @@
+MIT License
+
+Copyright (c) 2026 DocuSeal, LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+associated documentation files (the "Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the
+following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial
+portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
+LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO
+EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -24,12 +24,6 @@ Or install it yourself as:
 gem install rllama
 ```
 
-### Troubleshooting
-
-#### `llama_model_load_from_file_impl: no backends are loaded`
-
-If you're running on an Intel (x86_64) CPU and encounter this error while loading a model, make sure you're using the latest version of the gem. Rllama now preloads the bundled GGML backend libraries and automatically augments the `GGML_BACKEND_PATH`, so upgrading resolves the issue without any manual steps. If you build from source, ensure that directory is included in `GGML_BACKEND_PATH` before booting your Ruby process.
-
 ## CLI Chat
 
 The `rllama` command-line utility provides an interactive chat interface for conversing with language models. After installing the gem, you can start chatting immediately:
@@ -93,7 +87,14 @@ model.close
 
 #### Generation parameters
 
-Adjust the generation with parameters:
+By default the sampling parameters are taken from the model's own
+`general.sampling.*` GGUF metadata when present (falling back to llama.cpp's
+standard defaults: `temperature: 0.8`, `top_k: 40`, `top_p: 0.95`,
+`min_p: 0.05`). You can inspect the resolved values with
+`model.sampling_defaults`, or read any metadata key directly with
+`model.meta('general.sampling.temp')`.
+
+Pass any parameter explicitly to override the model default:
 
 ```ruby
 result = model.generate(
@@ -106,6 +107,9 @@ result = model.generate(
 )
 ```
 
+The parameters actually used are reported back in `result.stats`
+(`temperature`, `top_k`, `top_p`, `min_p`, `seed`).
+
 #### Streaming generation
 
 Stream generated text token-by-token:
@@ -141,6 +145,95 @@ result = model.generate([
 puts result.text
 ```
 
+#### Tool definitions
+
+Pass tool definitions to include them in the model's prompt for function calling:
+
+```ruby
+tools = [
+  {
+    type: 'function',
+    function: {
+      name: 'get_weather',
+      description: 'Get the current weather for a location',
+      parameters: {
+        type: 'object',
+        properties: {
+          location: { type: 'string', description: 'City name' }
+        },
+        required: ['location']
+      }
+    }
+  }
+]
+
+result = model.generate('What is the weather in Paris?', tools:)
+puts result.text
+```
+
+#### Tool call results
+
+Messages are passed to the model's chat template in the OpenAI format, so tool
+calls and their results are wired through the standard message schema: the
+assistant message carries a structured `tool_calls:` array, and the tool output
+is sent back as a `role: 'tool'` message.
+
+Pass `tools:` (and optionally `system:`) once when creating the context — they
+are remembered for all subsequent turns. When the model emits a tool call, it
+is parsed into `result.tool_calls` and stored in the conversation history in
+structured form automatically. Execute the calls and send each result back as
+a `role: 'tool'` message:
+
+```ruby
+context = model.init_context(tools:)
+
+result = context.generate('What is the weather in Paris?')
+result.tool_calls
+# => [{ name: 'get_weather', arguments: { 'location' => 'Paris' }, id: 'call_1' }]
+
+result.tool_calls.each do |call|
+  weather = get_weather(call[:arguments]['location'])
+
+  result = context.generate(
+    { role: 'tool', tool_call_id: call[:id], name: call[:name], content: weather.to_json }
+  )
+end
+
+puts result.text
+# => "The weather in Paris right now is 21°C."
+```
+
+The tool call markup is parsed by llama.cpp's chat parser, which derives the
+format automatically from the model's own chat template — any model with a
+tool-capable template is supported. If no tool call is detected,
+`result.tool_calls` is empty and `result.text` contains the raw output.
+
+The context detects the edited history and automatically rolls the KV cache
+back to the common prefix, so only the changed suffix is re-processed.
+
+### Reasoning
+
+For models with reasoning support, pass `reasoning: true` to
+enable the model's reasoning phase. The reasoning block is parsed out of the
+output: `result.text` contains only the final answer and the reasoning is
+available separately:
+
+```ruby
+result = model.generate('Why is the sky blue?', reasoning: true)
+
+result.reasoning
+# => "Thinking Process:\n\n1. Analyze the request..."
+
+result.text
+# => "The sky appears blue because..."
+```
+
+Like `tools:`, the flag can also be set once per context via
+`model.init_context(reasoning: true)`. It defaults to off; `result.reasoning`
+is `nil` when reasoning is disabled or the model produced no reasoning block.
+Note that when streaming with a block, the raw reasoning tokens are yielded
+as they are generated.
+
 ### Chat
 
 For ongoing conversations, use a context object that maintains the conversation history:
@@ -213,7 +306,6 @@ You can download GGUF format models from various sources:
 
 - [Hugging Face](https://huggingface.co/models?library=gguf) - Search for models with "GGUF" format
 
-
 ## License
 
 MIT

data/lib/rllama/common.rb

@@ -0,0 +1,134 @@
+# frozen_string_literal: true
+
+require 'ffi'
+require 'json'
+
+module Rllama
+  module Common
+    extend FFI::Library
+
+    REASONING_FORMAT_AUTO = 1
+
+    lib_file =
+      case FFI::Platform::OS
+      when 'darwin'
+        'libllama_common.dylib'
+      when 'windows', 'mingw32'
+        'llama_common.dll'
+      else
+        'libllama_common.so'
+      end
+
+    ffi_lib File.join(Cpp::PLATFORM_DIR, lib_file)
+
+    attach_function :rlc_free, [:pointer], :void
+    attach_function :rlc_chat_templates_init, %i[pointer string string string pointer], :pointer
+    attach_function :rlc_chat_templates_free, [:pointer], :void
+    attach_function :rlc_chat_apply, %i[pointer string pointer], :pointer
+    attach_function :rlc_chat_parse, %i[string bool string pointer], :pointer
+    attach_function :rlc_sampler_init, %i[pointer string pointer], :pointer
+    attach_function :rlc_sampler_free, [:pointer], :void
+    attach_function :rlc_sampler_sample, %i[pointer pointer int32], :int32
+    attach_function :rlc_sampler_accept, %i[pointer int32 bool], :void
+    attach_function :rlc_sampler_reset, [:pointer], :void
+    attach_function :rlc_sampler_print, [:pointer], :pointer
+
+    module_function
+
+    def chat_templates_init(model, source, bos_token, eos_token)
+      with_err do |err|
+        handle = rlc_chat_templates_init(model, source, bos_token, eos_token, err)
+
+        raise Error, "Failed to parse chat template: #{read_err(err)}" if handle.null?
+
+        handle
+      end
+    end
+
+    def chat_templates_free(handle)
+      rlc_chat_templates_free(handle)
+    end
+
+    def chat_apply(handle, messages:, tools: nil, add_generation_prompt: true, enable_thinking: false,
+                   add_bos: false, add_eos: false)
+      inputs = { messages:, tools:, add_generation_prompt:, enable_thinking:, add_bos:, add_eos:,
+                 reasoning_format: enable_thinking ? REASONING_FORMAT_AUTO : 0 }.to_json
+
+      with_err do |err|
+        result = rlc_chat_apply(handle, inputs, err)
+
+        raise Error, "Failed to apply chat template: #{read_err(err)}" if result.null?
+
+        JSON.parse(take(result))
+      end
+    end
+
+    def chat_parse(text, params, is_partial: false, reasoning: false)
+      parse_params = params.slice('format', 'parser', 'generation_prompt')
+      parse_params['reasoning_format'] = REASONING_FORMAT_AUTO if reasoning
+      params_json = parse_params.to_json
+
+      with_err do |err|
+        result = rlc_chat_parse(text, is_partial, params_json, err)
+
+        raise Error, "Failed to parse output: #{read_err(err)}" if result.null?
+
+        JSON.parse(take(result))
+      end
+    end
+
+    def sampler_init(model, params)
+      with_err do |err|
+        handle = rlc_sampler_init(model, params.to_json, err)
+
+        raise Error, "Failed to initialize sampler: #{read_err(err)}" if handle.null?
+
+        handle
+      end
+    end
+
+    def sampler_free(handle)
+      rlc_sampler_free(handle)
+    end
+
+    def sampler_sample(handle, ctx, idx = -1)
+      rlc_sampler_sample(handle, ctx, idx)
+    end
+
+    def sampler_accept(handle, token, accept_grammar: true)
+      rlc_sampler_accept(handle, token, accept_grammar)
+    end
+
+    def sampler_reset(handle)
+      rlc_sampler_reset(handle)
+    end
+
+    def sampler_print(handle)
+      take(rlc_sampler_print(handle))
+    end
+
+    def with_err
+      yield FFI::MemoryPointer.new(:pointer)
+    end
+
+    def take(ptr)
+      str = ptr.read_string
+
+      rlc_free(ptr)
+
+      str
+    end
+
+    def read_err(err)
+      msg_ptr = err.read_pointer
+
+      return 'unknown error' if msg_ptr.null?
+
+      msg = msg_ptr.read_string
+
+      rlc_free(msg_ptr)
+
+      msg
+    end
+  end
+end

data/lib/rllama/context.rb

@@ -1,13 +1,20 @@
 # frozen_string_literal: true
 
 require 'etc'
+require 'json'
 
 module Rllama
   class Context
+    TOOL_CALL_ID_PREFIX = 'call_'
+    GRAMMAR_TYPE_TOOL_CALLS = 3
+
     attr_reader :messages, :n_ctx, :n_batch, :n_past
 
-    def initialize(model, embeddings: false, n_ctx: nil, n_batch: nil, n_threads: Etc.nprocessors)
+    def initialize(model, embeddings: false, n_ctx: nil, n_batch: nil, n_threads: Etc.nprocessors,
+                   system: nil, tools: nil, reasoning: false)
       @model = model
+      @tools = tools
+      @reasoning = reasoning
       @n_ctx = n_ctx
       @n_batch = n_batch
       @embeddings = embeddings
@@ -38,13 +45,32 @@ module Rllama
 
       raise Error, 'Failed to create the llama_context' if @pointer.null?
 
+      @n_ctx = Cpp.llama_n_ctx(@pointer)
+      @n_batch = Cpp.llama_n_batch(@pointer)
+
       @n_past = 0
+      @cache_tokens = []
+      @tool_call_count = 0
       @messages = []
+      @messages << { role: 'system', content: system } if system
+
+      prefill if !@embeddings && (system || tools)
     end
 
-    def generate(message, role: 'user', max_tokens: @n_ctx, temperature: 0.8, top_k: 40, top_p: 0.95, min_p: 0.05,
-                 seed: nil, system: nil)
-      @messages << { role: 'system', content: system } if system && @messages.empty?
+    def generate(message, role: 'user', max_tokens: @n_ctx, temperature: nil, top_k: nil, top_p: nil, min_p: nil,
+                 seed: nil, system: nil, tools: nil, reasoning: nil, &block)
+      temperature, top_k, top_p, min_p = resolve_sampling(temperature, top_k, top_p, min_p)
+
+      @tools = tools unless tools.nil?
+      @reasoning = reasoning unless reasoning.nil?
+
+      if system
+        if @messages.dig(0, :role).to_s == 'system'
+          @messages.first[:content] = system
+        else
+          @messages.unshift(role: 'system', content: system)
+        end
+      end
 
       if message.is_a?(Array)
         @messages.push(*message)
@@ -54,51 +80,19 @@ module Rllama
         @messages << { role: role, content: message }
       end
 
-      prompt_string = @model.build_chat_template(@messages)
-
-      n_prompt_tokens = -Cpp.llama_tokenize(@model.vocab, prompt_string, prompt_string.bytesize, nil, 0, true, true)
-
-      raise Error, 'Prompt is too long.' if n_prompt_tokens.negative?
-
-      prompt_tokens_ptr = FFI::MemoryPointer.new(:int32, n_prompt_tokens)
-      tokens_written = Cpp.llama_tokenize(@model.vocab, prompt_string, prompt_string.bytesize, prompt_tokens_ptr,
-                                          n_prompt_tokens, true, true)
-
-      raise Error, 'Failed to tokenize prompt.' if tokens_written.negative?
-
-      new_token_count = tokens_written - @n_past
-
-      if new_token_count.positive?
-        new_tokens_ptr = prompt_tokens_ptr + (@n_past * FFI.type_size(:int32))
-
-        batch = Cpp.llama_batch_get_one(new_tokens_ptr, new_token_count)
-
-        raise Error, 'llama_decode failed.' if Cpp.llama_decode(@pointer, batch) != 0
-
-        @n_past = tokens_written
-      end
-
-      chain_params = Cpp.llama_sampler_chain_default_params
-      sampler_chain = Cpp.llama_sampler_chain_init(chain_params)
+      applied = @model.apply_chat_template(@messages, tools: @tools, enable_thinking: @reasoning)
 
-      Cpp.llama_sampler_chain_add(sampler_chain, Cpp.llama_sampler_init_min_p(min_p, 1)) if min_p
-      Cpp.llama_sampler_chain_add(sampler_chain, Cpp.llama_sampler_init_top_k(top_k)) if top_k&.positive?
-      Cpp.llama_sampler_chain_add(sampler_chain, Cpp.llama_sampler_init_top_p(top_p, 1)) if top_p && top_p < 1.0
-      if temperature&.positive?
-        Cpp.llama_sampler_chain_add(sampler_chain,
-                                    Cpp.llama_sampler_init_temp(temperature))
-      end
+      decode_prompt(applied['prompt'])
 
-      is_probabilistic = temperature&.positive? || top_k&.positive? || (top_p && top_p < 1.0) || !min_p.nil?
       rng_seed = seed || (Random.new_seed & 0xFFFFFFFF)
 
-      if is_probabilistic
-        Cpp.llama_sampler_chain_add(sampler_chain, Cpp.llama_sampler_init_dist(rng_seed))
-      else
-        Cpp.llama_sampler_chain_add(sampler_chain, Cpp.llama_sampler_init_greedy)
-      end
+      sampler = Common.sampler_init(@model.pointer, sampler_params(temperature, top_k, top_p, min_p, rng_seed, applied))
+
+      stops = applied['additional_stops'].to_a.map(&:b)
+      max_stop = stops.map(&:bytesize).max || 0
 
       n_decoded = 0
+      n_yielded = 0
 
       generated_text = ''.b
 
@@ -111,7 +105,9 @@ module Rllama
       loop do
         break if n_decoded >= max_tokens
 
-        new_token_id = Cpp.llama_sampler_sample(sampler_chain, @pointer, -1)
+        new_token_id = Common.sampler_sample(sampler, @pointer)
+
+        Common.sampler_accept(sampler, new_token_id)
 
         break if Cpp.llama_vocab_is_eog(@model.vocab, new_token_id)
 
@@ -119,10 +115,10 @@ module Rllama
         n_chars = Cpp.llama_token_to_piece(@model.vocab, new_token_id, buffer, buffer.size, 0, true)
 
         if n_chars >= 0
-          piece_bytes = buffer.read_string(n_chars)
-          utf8_piece = piece_bytes.force_encoding(Encoding::UTF_8)
-          generated_text << utf8_piece
-          yield utf8_piece if block_given?
+          stop_at, n_yielded = emit_piece(generated_text, buffer.read_string(n_chars), stops, max_stop, n_yielded,
+                                          &block)
+
+          break if stop_at
         end
 
         token_ptr = FFI::MemoryPointer.new(:int32, 1).put_int32(0, new_token_id)
@@ -132,24 +128,54 @@ module Rllama
         raise Error, 'llama_decode failed.' if Cpp.llama_decode(@pointer, batch) != 0
 
         @n_past += 1
+        @cache_tokens << new_token_id
         n_decoded += 1
       end
 
+      if block_given? && n_yielded < generated_text.bytesize
+        yield generated_text.byteslice(n_yielded..).force_encoding(Encoding::UTF_8)
+      end
+
+      generated_text.force_encoding(Encoding::UTF_8)
+
       end_time = Time.now
 
       duration = end_time - start_time
 
       tps = n_decoded.positive? && duration.positive? ? n_decoded / duration : 0
 
-      Cpp.llama_sampler_free(sampler_chain)
+      Common.sampler_free(sampler)
+
+      text = generated_text
+      reasoning = nil
+      tool_calls = []
+
+      if @tools || @reasoning
+        parsed = Common.chat_parse(generated_text, applied, reasoning: @reasoning)
+
+        reasoning = parsed['reasoning_content']
+        reasoning = nil if reasoning&.empty?
+        tool_calls = extract_tool_calls(parsed, assistant_message) if @tools
+
+        if @reasoning && tool_calls.empty?
+          text = parsed['content'].to_s
+          assistant_message[:content] = text
+        end
+      end
 
       Result.new(
-        text: generated_text,
+        text:,
+        reasoning:,
+        tool_calls:,
         stats: {
           duration:,
           tokens_generated: n_decoded,
           tps:,
-          seed: rng_seed
+          seed: rng_seed,
+          temperature:,
+          top_k:,
+          top_p:,
+          min_p:
         }
       )
     end
@@ -248,6 +274,149 @@ module Rllama
       Cpp.llama_free(@pointer)
     end
 
+    def prefill
+      decode_prompt(@model.build_chat_template(@messages, tools: @tools, add_generation_prompt: false,
+                                                          enable_thinking: @reasoning))
+    rescue StandardError
+      nil
+    end
+
+    def extract_tool_calls(parsed, assistant_message)
+      calls = (parsed['tool_calls'] || []).map do |call|
+        id = call['id'].to_s
+        id = "#{TOOL_CALL_ID_PREFIX}#{@tool_call_count += 1}" if id.empty?
+
+        arguments = begin
+          JSON.parse(call.dig('function', 'arguments').to_s)
+        rescue JSON::ParserError
+          {}
+        end
+
+        { name: call.dig('function', 'name'), arguments:, id: }
+      end
+
+      return [] if calls.empty?
+
+      assistant_message.replace(
+        role: 'assistant',
+        content: nil,
+        tool_calls: calls.map do |call|
+          { type: 'function', id: call[:id],
+            function: { name: call[:name], arguments: call[:arguments].to_json } }
+        end
+      )
+
+      calls
+    end
+
+    def decode_prompt(prompt_string)
+      n_prompt_tokens = -Cpp.llama_tokenize(@model.vocab, prompt_string, prompt_string.bytesize, nil, 0, true, true)
+
+      raise Error, 'Prompt is too long.' if n_prompt_tokens.negative?
+
+      prompt_tokens_ptr = FFI::MemoryPointer.new(:int32, n_prompt_tokens)
+      tokens_written = Cpp.llama_tokenize(@model.vocab, prompt_string, prompt_string.bytesize, prompt_tokens_ptr,
+                                          n_prompt_tokens, true, true)
+
+      raise Error, 'Failed to tokenize prompt.' if tokens_written.negative?
+
+      prompt_tokens = prompt_tokens_ptr.read_array_of_int32(tokens_written)
+
+      common = common_prefix_length(prompt_tokens)
+
+      if common < @cache_tokens.length
+        memory = Cpp.llama_get_memory(@pointer)
+
+        if memory.null? || !Cpp.llama_memory_seq_rm(memory, 0, common, -1)
+          Cpp.llama_memory_clear(memory, true) unless memory.null?
+
+          common = 0
+        end
+
+        @n_past = common
+      end
+
+      while tokens_written > @n_past
+        n_eval = [tokens_written - @n_past, @n_batch].min
+
+        new_tokens_ptr = prompt_tokens_ptr + (@n_past * FFI.type_size(:int32))
+
+        batch = Cpp.llama_batch_get_one(new_tokens_ptr, n_eval)
+
+        raise Error, 'llama_decode failed.' if Cpp.llama_decode(@pointer, batch) != 0
+
+        @n_past += n_eval
+      end
+
+      @cache_tokens = prompt_tokens
+    end
+
+    def common_prefix_length(tokens)
+      limit = [@cache_tokens.length, tokens.length - 1].min
+
+      common = 0
+      common += 1 while common < limit && @cache_tokens[common] == tokens[common]
+
+      common
+    end
+
+    def sampler_params(temperature, top_k, top_p, min_p, seed, applied)
+      params = {
+        temp: temperature || 0.0,
+        top_k: top_k&.positive? ? top_k : 0,
+        top_p: top_p || 1.0,
+        min_p: min_p || 0.0,
+        seed:
+      }
+
+      grammar = applied['grammar'].to_s
+
+      unless grammar.empty?
+        params.merge!(grammar:, grammar_type: GRAMMAR_TYPE_TOOL_CALLS,
+                      grammar_lazy: applied['grammar_lazy'],
+                      grammar_triggers: applied['grammar_triggers'],
+                      generation_prompt: applied['generation_prompt'])
+      end
+
+      params
+    end
+
+    def emit_piece(text, piece, stops, max_stop, n_yielded, &block)
+      tail_from = [text.bytesize - max_stop + 1, 0].max
+      text << piece
+
+      stop_at = max_stop.positive? ? stops.filter_map { |stop| text.index(stop, tail_from) }.min : nil
+      text.slice!(stop_at..) if stop_at
+
+      if block
+        safe = text.bytesize - (stop_at ? 0 : stop_holdback(text, stops))
+
+        if safe > n_yielded
+          yield(text.byteslice(n_yielded, safe - n_yielded).force_encoding(Encoding::UTF_8))
+          n_yielded = safe
+        end
+      end
+
+      [stop_at, n_yielded]
+    end
+
+    def stop_holdback(text, stops)
+      stops.map do |stop|
+        (stop.bytesize - 1).downto(1).find { |n| text.end_with?(stop.byteslice(0, n)) } || 0
+      end.max || 0
+    end
+
+    def resolve_sampling(temperature, top_k, top_p, min_p)
+      defaults = @model.sampling_defaults
+
+      [
+        temperature.nil? ? defaults[:temperature] : temperature,
+        top_k.nil? ? defaults[:top_k] : top_k,
+        top_p.nil? ? defaults[:top_p] : top_p,
+        min_p.nil? ? defaults[:min_p] : min_p
+      ]
+    end
+
     def norm(vec)
       Math.sqrt(vec.sum { |x| x**2 })
     end