llm.rb 15.2.2 → 15.4.0

data/docs/deepdive/reference/tracer.md

@@ -134,6 +134,82 @@ OpenTelemetry.
 The tracer can also write to a file with the `path:` option to
 [`LLM::Tracer::Logger.new`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer/Logger.html#initialize-instance_method).
 
+### Hooks
+
+#### Overview
+
+[`LLM::Tracer`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html)
+exposes one method per event in a request's lifecycle. A subclass
+implements the events it cares about and routes them anywhere: a
+logger, a metrics counter, or a database table.
+
+#### How it works
+
+Three hooks cover a provider request. `on_request_start` fires before
+the request is sent and returns the span that `on_request_finish` and
+`on_request_error` receive. Every request carries a `request_id`, a
+UUIDv7 minted when the request begins and passed to all three hooks
+for that request, so a tracer can correlate its events even when a
+turn makes several requests.
+
+Three more hooks cover a local tool call. `on_tool_start` fires before
+the tool runs and returns the span that `on_tool_finish` and
+`on_tool_error` receive.
+
+A turn is additionally bracketed with `start_trace` and `stop_trace`.
+The runtime calls them around every agent turn with a `trace_group_id`,
+and a tracer that supports it (such as
+[`LLM::Tracer::Telemetry`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer/Telemetry.html))
+uses that id to give every span of the turn the same trace id:
+
+```ruby
+class MyTracer < LLM::Tracer
+  def on_request_start(operation:, model: nil, **)
+    warn "start #{operation} #{model}"
+  end
+
+  def on_request_finish(operation:, res:, **)
+    warn "finish #{operation}"
+  end
+
+  def on_request_error(ex:, **)
+    warn "error #{ex.class}"
+  end
+
+  def on_tool_start(id:, name:, arguments:, model:, **)
+    warn "tool #{name}"
+  end
+
+  def on_tool_finish(result:, **)
+    warn "tool #{result.name} done"
+  end
+
+  def on_tool_error(ex:, **)
+    warn "tool error #{ex.class}"
+  end
+end
+
+llm = LLM.deepseek(key: ENV["KEY"])
+llm.tracer = MyTracer.new(llm)
+agent = LLM::Agent.new(llm)
+agent.talk "Hello"
+```
+
+#### Why would I use it?
+
+The built-in tracers cover logging and OpenTelemetry. A hook lets you
+send the same events somewhere else, and because the built-in tracers
+accept the keywords they do not use, existing tracer code keeps working
+as hooks gain parameters.
+
+#### Notes
+
+The base class raises `NotImplementedError` for any hook it does not
+implement, so a tracer must cover every hook the runtime calls: the six
+request and tool hooks above. Accept `**` to absorb keywords you do not
+read, as the built-in tracers do, so a hook that gains a parameter does
+not break your subclass.
+
 ### PrettyLogger
 
 #### Overview

data/docs/deepdive.md

@@ -44,7 +44,7 @@ useful when you need to go beyond the basics.
 #### Notes
 
 The deepdive is a living document. Sections are added as new
-features land. The [README.md](https://github.com/r-uby-dev/llm#readme)
+features land. The [README.md](https://github.com/r-uby-dev/llm.rb#readme)
 is the best place to start if you are new to llm.rb.
 
 ---

data/lib/llm/active_record/message.rb

@@ -0,0 +1,113 @@
+# frozen_string_literal: true
+
+module LLM::ActiveRecord
+  ##
+  # Represents a message in an agent's memory.
+  #
+  # This class is virtual and never materializes
+  # in the database as a real table. It provides
+  # a SQL view into the messages array stored in
+  # the JSONB column that carries an agent's
+  # runtime state, and it returns relations, so
+  # messages can be filtered and ordered in the
+  # database instead of in memory.
+  #
+  # It expects the agent and this class to share a
+  # connection, which they do when both live on the
+  # same database - the default in Rails.
+  #
+  # @example
+  #   LLM::ActiveRecord::Message.for(agent:)
+  #                             .where(role: "assistant")
+  #                             .count
+  class Message < ActiveRecord::Base
+    ##
+    # The name the derived table is given. It is "created"
+    # on-demand and it is here so that ActiveRecord has
+    # something to select from.
+    self.table_name = "llm_agent_messages"
+
+    ##
+    # The derived table does not exist, so there is no schema
+    # to load. Without this, ActiveRecord asks the database
+    # for the columns of a table that was never created.
+    # @return [void]
+    def self.load_schema!
+      @columns_hash = {}.freeze
+    end
+
+    ##
+    # Build a relation over one agent's messages.
+    #
+    # The table, the column and its type come from the
+    # agent's own class, so this works for any model that
+    # keeps its conversation in a column.
+    #
+    # The base fields the query guarantees are columns -
+    # id, role, content, tools - so they are what a
+    # caller's `where` and `order` are written against.
+    # The whole message is carried along as `data`, for
+    # fields the query does not name yet.
+    #
+    # The derived table is aliased as `llm_agent_messages`
+    # because ActiveRecord qualifies its SELECT with the
+    # class' table name.
+    #
+    # @param [ActiveRecord::Base] agent
+    #  An instance of an ActiveRecord model.
+    # @return [ActiveRecord::Relation]
+    def self.for(agent:)
+      klass      = agent.class
+      connection = klass.connection
+      table      = connection.quote_table_name(klass.table_name)
+      options    = klass.llm_plugin_options
+      column     = connection.quote_column_name(options.fetch(:data_column))
+      from(<<~SQL).where(agent_id: agent&.id)
+        (SELECT #{table}.id                AS agent_id,
+                (message ->> 'id')         AS id,
+                (message ->> 'role')       AS role,
+                (message ->> 'content')    AS content,
+                (message -> 'tools')       AS tools,
+                ordinality                 AS position,
+                message                    AS data
+         FROM #{table},
+              jsonb_array_elements(#{table}.#{column} -> 'messages')
+                WITH ORDINALITY AS each(message, ordinality))
+          AS llm_agent_messages
+      SQL
+    end
+
+    ##
+    # @return (see LLM::Message#tool_call?)
+    def tool_call?
+      unwrap!.tool_call?
+    end
+
+    ##
+    # @return (see LLM::Message#tool_return?)
+    def tool_return?
+      unwrap!.tool_return?
+    end
+
+    ##
+    # The message, as the runtime would hand it back.
+    #
+    # The whole serialized message is used, so fields the
+    # query does not name: usage, reasoning content,
+    # and compaction survive the round trip through the
+    # database.
+    #
+    # @return [LLM::Message]
+    def unwrap!
+      @message ||= begin
+        stored = data.is_a?(Hash) ? data : {}
+        extra = stored.each_with_object({}) { |(key, value), acc| acc[key.to_sym] = value }
+        extra[:id] = stored["id"] || id
+        extra[:role] ||= role
+        extra[:content] ||= content
+        extra[:tool_calls] = stored["tools"] || tools
+        LLM::Message.new(extra[:role], extra[:content], extra)
+      end
+    end
+  end
+end

data/lib/llm/active_record.rb

@@ -74,4 +74,5 @@ module LLM::ActiveRecord
 
   require "llm/active_record/acts_as_llm"
   require "llm/active_record/acts_as_agent"
+  require "llm/active_record/message"
 end

data/lib/llm/agent.rb

@@ -17,10 +17,6 @@ module LLM
   # **Notes:**
   # * Instructions are injected once unless a system message is already present.
   # * An agent automatically executes tool loops (unlike {LLM::Context LLM::Context}).
-  # * The automatic tool loop enables the wrapped context's `guard` by default.
-  #   The built-in {LLM::Guard::Loop LLM::Guard::Loop} detects repeated
-  #   tool-call patterns and blocks stuck execution before more tool work is
-  #   queued.
   # * The tool loop can be bounded with `tool_budget`. Once the budget is
   #   spent, no further tool calls are run for that turn: the agent sends an
   #   in-band advisory message back through the model instead, and keeps
@@ -106,6 +102,8 @@ module LLM
     #   end
     #
     # @param [Hash] properties
+    # @option properties [String, Symbol, Proc] :name
+    # @option properties [String, Symbol, Proc] :description
     # @option properties [String] :instructions
     # @option properties [String] :model
     # @option properties [Array<LLM::Function>] :tools
@@ -138,7 +136,7 @@ module LLM
     # @return [String]
     #  Return's the agents name
     def self.name(name = UNDEFINED, &block)
-      if name.equal?(UNDEFINED)
+      if name.equal?(UNDEFINED) and block.nil?
         if @name.nil?
           name  = to_s.split("::").last
           @name = name.gsub(CASE_PATTERN, "-").downcase
@@ -158,6 +156,11 @@ module LLM
     ##
     # Set or get an agent's description
     # @note
+    #  Reading this on the class returns what was configured - a
+    #  Symbol or Proc included - because an instance is what
+    #  resolves those. Read it on an instance for the description
+    #  itself.
+    # @note
     #  This method serves as a self-documenting string.
     #  It is optional but recommended.
     # @param [String] desc
@@ -165,7 +168,7 @@ module LLM
     # @return [String, nil]
     #  Returns the agent's description
     def self.description(desc = UNDEFINED, &block)
-      if desc.equal?(UNDEFINED)
+      if desc.equal?(UNDEFINED) and block.nil?
         @desc
       else
         @desc = block || desc
@@ -338,10 +341,10 @@ module LLM
     #  The path to a file
     # @return [String, nil]
     def self.path(path = UNDEFINED, &block)
-      if path.equal?(UNDEFINED)
+      if path.equal?(UNDEFINED) and block.nil?
         @path
       else
-        @path = path || block
+        @path = block || path
       end
     end
 
@@ -365,10 +368,10 @@ module LLM
     #  a single turn.
     # @return [Integer, nil]
     def self.tool_budget(budget = UNDEFINED, &block)
-      if budget.equal?(UNDEFINED)
+      if budget.equal?(UNDEFINED) and block.nil?
         @tool_budget
       else
-        @tool_budget = budget || block
+        @tool_budget = block || budget
       end
     end
 
@@ -434,13 +437,11 @@ module LLM
         end
       end
       ##
-      # Alibaba (token plan) will frequently issue rate
-      # limits or time outs that it recovers from. The
-      # higher retry count is to account for scenarios
-      # where it takes longer than expected to recover.
-      retry_budget = llm.name == :alibaba ? 8 : 5
-      params[:retry_budget] = retry_budget if params[:retry_budget].equal?(UNDEFINED)
-      @ctx = LLM::Context.new(llm, {guard: LLM::Guard::Loop}.merge(params))
+      # The provider decides how patient a turn should be: it
+      # knows its own API, and how often it recovers from a
+      # rate limit rather than failing outright.
+      params[:retry_budget] = llm.retry_budget if params[:retry_budget].equal?(UNDEFINED)
+      @ctx = LLM::Context.new(llm, params)
       @path and File.readable?(@path) ? @ctx.restore(path:) : nil
     end
 
@@ -510,6 +511,22 @@ module LLM
       @ctx.messages
     end
 
+    ##
+    # Returns a stable id for this agent. The id is borrowed from
+    # the context it wraps, so it survives save and restore.
+    # @return [String]
+    def id
+      @ctx.id
+    end
+
+    ##
+    # Returns the time this agent was created. It is derived
+    # from the context id, so it is not persisted separately.
+    # @return [Time, nil]
+    def created_at
+      @ctx.created_at
+    end
+
     ##
     # @return [Integer]
     def retry_budget
@@ -851,8 +868,16 @@ module LLM
         end
         res
       end
-      return run.call unless @tracer
-      @llm.with_tracer(@tracer, &run)
+      ##
+      # One turn, one trace group. A tracer
+      # is told where the turn begins and
+      # where it ends. The trace group ID
+      # identifies the turn.
+      tracer = @tracer || @llm.tracer
+      tracer.start_trace(name: "llm.turn", trace_group_id: SecureRandom.uuid_v7)
+      @llm.with_tracer(tracer, &run)
+    ensure
+      tracer&.stop_trace
     end
 
     ##

data/lib/llm/console/buffer.rb

@@ -72,6 +72,14 @@ class LLM::Console
       @snapshot = nil
     end
 
+    ##
+    # Returns true while the buffer has an active row
+    # for the streaming path to replace.
+    # @return [Boolean]
+    def open?
+      !@snapshot.nil?
+    end
+
     ##
     # @return [void]
     def scroll_up(height)
@@ -131,7 +139,7 @@ class LLM::Console
     #  One or more chunks.
     # @return [void]
     def replace(chunks)
-      @rows = @snapshot.map(&:dup)
+      @rows = @snapshot ? @snapshot.map(&:dup) : @rows
       chunks.each { wrap(_1, @rows) }
     end
 

data/lib/llm/console.rb

@@ -284,6 +284,11 @@ module LLM
           # UI can repaint. Cap the stream chunks drained per call so
           # read! gives control back to the key loop: leftover chunks
           # stay queued and are drained by the next read!.
+          #
+          # A turn that was cancelled can still have chunks queued, and
+          # they arrive after the buffer is closed. There is no active
+          # row to replace at that point, so they are dropped.
+          next unless buffer.open?
           status.text = think_text if stream.tools.empty?
           write_message name, markdown(value), method: :replace
           burst += 1
@@ -292,7 +297,7 @@ module LLM
           self.status = value
         when :done
           status.text = "idle"
-          write_message name, markdown(value), method: :replace
+          write_message(name, markdown(value), method: :replace) if buffer.open?
           buffer.close
           @thread = nil
         when :cancel

data/lib/llm/context/deserializer.rb

@@ -5,7 +5,12 @@ class LLM::Context
   # @api private
   module Deserializer
     ##
-    # Restore a saved context state
+    # Restore a saved context state.
+    #
+    # The `context_used` and `context_window` keys carried by a payload
+    # are projections for queryability. They are deliberately ignored
+    # here: the runtime derives both from the messages and the
+    # registry, so a payload can never seed them.
     # @param [String, nil] path
     #  The path to a JSON file
     # @param [String, nil] string
@@ -25,6 +30,8 @@ class LLM::Context
       else
         LLM.json.load(string)
       end
+      @id = ctx["id"] || @id
+      @created_at = nil
       @messages.concat [*ctx["messages"]].map { deserialize_message(_1) }
       @compacted = !!ctx["compacted"]
       self
@@ -40,8 +47,8 @@ class LLM::Context
       usage = payload["usage"]
       reasoning_content = payload["reasoning_content"]
       compaction = payload["compaction"]
-      created_at = payload["created_at"]
-      extra = {tool_calls:, original_tool_calls:, tools: @params[:tools], usage:, reasoning_content:, compaction:, created_at:}.compact
+      id = payload["id"]
+      extra = {tool_calls:, original_tool_calls:, tools: @params[:tools], usage:, reasoning_content:, compaction:, id:}.compact
       content = returns.nil? ? deserialize_content(payload["content"]) : returns
       LLM::Message.new(payload["role"], content, extra)
     end

data/lib/llm/context.rb

@@ -55,7 +55,7 @@ module LLM
     # @api private
     # @return [Array<Symbol>]
     def self.params
-      %w[guard retry_budget concurrency transformer compactor record]
+      %w[guard retry_budget concurrency transformer compactor record id]
     end
 
     ##
@@ -78,6 +78,12 @@ module LLM
     # @return [Object, nil]
     attr_reader :record
 
+    ##
+    # Returns a stable id for this context. It is generated once
+    # on creation and restored with the runtime state on load.
+    # @return [String]
+    attr_reader :id
+
     ##
     # @param [LLM::Provider] llm
     #  A provider
@@ -89,6 +95,8 @@ module LLM
     #   Defaults to `:responses` for OpenAI, otherwise it defaults
     #   to `:completions`.
     # @option params [String] :model Defaults to the provider's default model
+    # @option params [String] :id
+    #   A stable id for the context. Defaults to a UUID.
     # @option params [Class<LLM::Compactor>, nil] :compactor
     #   A compactor class to use for context compaction. Defaults to
     #   {LLM::Compactor::Null}.
@@ -109,6 +117,7 @@ module LLM
     def initialize(llm, params = {})
       params = {}.merge!(params)
       @llm = llm
+      @id = params.delete(:id) || SecureRandom.uuid_v7
       @record = params.delete(:record)
       @mode = params.delete(:mode) || (llm.name == :openai ? :responses : :completions)
       tools = [*params.delete(:tools), *load_skills(params.delete(:skills))]
@@ -174,10 +183,6 @@ module LLM
     # via {LLM::Stream#on_tool_call}. A blocked call yields its in-band
     # `guard_error` return without executing.
     #
-    # The built-in implementation is {LLM::Guard::Loop LLM::Guard::Loop}, which
-    # detects repeated tool-call patterns and turns them into in-band
-    # `guard_error` tool returns.
-    #
     # @return [Class<LLM::Guard>]
     def guard
       @guard[:klass]
@@ -489,12 +494,29 @@ module LLM
     end
 
     ##
+    # Returns the time this context was created, derived from the
+    # timestamp embedded in its UUIDv7 id, or nil when it is not.
+    # @return [Time, nil]
+    def created_at
+      @created_at ||= LLM::Utils.timestamp(@id)
+    end
+
+    ##
+    # Returns the runtime state as a Hash.
+    #
+    # `context_used` and `context_window` are projections written for
+    # queryability, so that saved state can be inspected at rest
+    # without loading it. They are never read back; the runtime
+    # always derives them from the messages and the registry.
     # @return [Hash]
     def to_h
       {
         schema_version: 1,
+        id: @id,
         model:,
         compacted:,
+        context_used:,
+        context_window:,
         messages: @messages.map { serialize_message(_1) }
       }
     end
@@ -597,33 +619,37 @@ module LLM
     # Executes a turn through the Responses API.
     # @api private
     def respond(prompt, params)
-      history = @messages.to_a
-      params = @params.merge(params).reject { self.class.params.include?(_1.to_s) }
-      extra = params.slice(:model, :tools).merge!(ctx: self, tracer:, guard: @guard[:klass].new(self))
-      params[:stream] = LLM::Stream.try(params[:stream], extra:)
-      res_id = params[:store] == false ? nil : @messages.find(&:assistant?)&.response&.response_id
-      input = res_id ? [] : history
-      params[:input] = input
-      messages = transform(prompt, params, key: :input)
-      @stream = params[:stream]
-      new_messages = messages[input.size..]
-      params = params.merge(previous_response_id: res_id, input:).compact
-      [new_messages, params, @llm.responses.create(messages, params)]
+      @llm.with(**headers) do
+        history = @messages.to_a
+        params = @params.merge(params).reject { self.class.params.include?(_1.to_s) }
+        extra = params.slice(:model, :tools).merge!(ctx: self, tracer:, guard: @guard[:klass].new(self))
+        params[:stream] = LLM::Stream.try(params[:stream], extra:)
+        res_id = params[:store] == false ? nil : @messages.find(&:assistant?)&.response&.response_id
+        input = res_id ? [] : history
+        params[:input] = input
+        messages = transform(prompt, params, key: :input)
+        @stream = params[:stream]
+        new_messages = messages[input.size..]
+        params = params.merge(previous_response_id: res_id, input:).compact
+        [new_messages, params, @llm.responses.create(messages, params)]
+      end
     end
 
     ##
     # Executes a turn through the chat completions API.
     # @api private
     def complete(prompt, params)
-      history = @messages.to_a
-      params = params.merge(messages: history)
-      params = @params.merge(params).reject { self.class.params.include?(_1.to_s) }
-      extra = params.slice(:model, :tools).merge!(ctx: self, tracer:, guard: @guard[:klass].new(self))
-      params[:stream] = LLM::Stream.try(params[:stream], extra:)
-      messages = transform(prompt, params)
-      @stream = params[:stream]
-      new_messages = messages[history.size..]
-      [new_messages, params, @llm.complete(messages, params)]
+      @llm.with(**headers) do
+        history = @messages.to_a
+        params = params.merge(messages: history)
+        params = @params.merge(params).reject { self.class.params.include?(_1.to_s) }
+        extra = params.slice(:model, :tools).merge!(ctx: self, tracer:, guard: @guard[:klass].new(self))
+        params[:stream] = LLM::Stream.try(params[:stream], extra:)
+        messages = transform(prompt, params)
+        @stream = params[:stream]
+        new_messages = messages[history.size..]
+        [new_messages, params, @llm.complete(messages, params)]
+      end
     end
 
     ##
@@ -652,5 +678,15 @@ module LLM
       end
       messages << LLM::Message.new(@llm.tool_role, cancelled) unless cancelled.empty?
     end
+
+    ##
+    # @return [Hash]
+    def headers
+      if @llm.name == :openrouter
+        {"x-session-id" => @id}
+      else
+        {}
+      end
+    end
   end
 end

data/lib/llm/guard.rb

@@ -10,16 +10,10 @@ module LLM
   # an {LLM::Function::Return LLM::Function::Return} that closes the
   # pending tool call, or `nil` when execution should continue. The guard
   # is stamped onto the functions the context binds, so it runs whenever a
-  # task is spawned — including tool calls queued from a stream. The
-  # built-in implementation is {LLM::Guard::Loop LLM::Guard::Loop}, which
-  # detects repeated tool-call patterns. {LLM::Guard::Null LLM::Guard::Null}
-  # is a no-op and the default.
-  #
-  # {LLM::Agent LLM::Agent} enables {LLM::Guard::Loop LLM::Guard::Loop}
-  # by default through its wrapped context.
+  # task is spawned — including tool calls queued from a stream.
+  # {LLM::Guard::Null LLM::Guard::Null} is a no-op and the default.
   class Guard
     require_relative "guard/null"
-    require_relative "guard/loop"
 
     ##
     # @return [LLM::Context]

data/lib/llm/message.rb

@@ -18,9 +18,10 @@ module LLM
     attr_reader :extra
 
     ##
-    # Returns the time the message was created
-    # @return [Time]
-    attr_reader :created_at
+    # Returns a stable id for this message. It is generated once
+    # on creation and restored with the runtime state on load.
+    # @return [String]
+    attr_reader :id
 
     ##
     # Returns a new message
@@ -32,7 +33,15 @@ module LLM
       @role = role.to_s
       @content = content
       @extra = LLM::Object.from(extra)
-      @created_at = extra[:created_at] ? Time.iso8601(extra[:created_at].to_s) : Time.now.utc
+      @id = extra[:id] || SecureRandom.uuid_v7
+    end
+
+    ##
+    # Returns the time this message was created, derived from the
+    # timestamp embedded in its UUIDv7 id, or nil when it is not.
+    # @return [Time, nil]
+    def created_at
+      @created_at ||= LLM::Utils.timestamp(@id)
     end
 
     ##
@@ -40,10 +49,10 @@ module LLM
     # @return [Hash]
     def to_h
       {
+        id: @id,
         role:,
         content:,
         reasoning_content:,
-        created_at: created_at.utc.iso8601,
         compaction: extra.compaction,
         tools: extra.tool_calls&.map { LLM::Object === _1 ? _1.to_h : _1 },
         usage:,
@@ -58,13 +67,15 @@ module LLM
     end
 
     ##
-    # Returns true when two objects have the same role and content
+    # Returns true when two objects have the same role and content.
+    # The id is ignored, because it identifies a message rather than
+    # describing it.
     # @param [Object] other
     #  The other object to compare
     # @return [Boolean]
     def ==(other)
       if other.respond_to?(:to_h)
-        to_h == other.to_h
+        to_h.except(:id) == other.to_h.except(:id)
       else
         false
       end