llm.rb 15.4.1 → 15.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4647447cce1ccfc7e4f062cfa46b6dd0363ffd1a86537200a7877daff983cda3
-  data.tar.gz: 31502899d4a6443b893a48baf18c75d02311920166a6d791d671fa33600e93b9
+  metadata.gz: df1a9b7fbf194f568b95f3f1b2d0314bb703027bc89e4432109d118f238ea2ba
+  data.tar.gz: 679938a2f6538f9c1999d31870462498393e1b3a7c7203630c5e8706c893e9ce
 SHA512:
-  metadata.gz: ee244a4b0d7d317632e1ee2f3dc44b82926086a71f2ded6ba7e3189eff9636dba2eed5f97d4ed6c83140584d3807b4872303a20b369a123be2336c32671758a1
-  data.tar.gz: 7ed45ea9770afabb6522b6939c9d333c28a5892e40e32de271d03a8c5fb3f9fa15b5c2ace19e74ac80342bc65e04a7b5edb48b6799a612f5754b8ccccd33bdaf
+  metadata.gz: 49151bad60a22a06b119e08062e58c6d35727d59d10e07d9921166a23b5817083ea11fe942a090bc0e64b68a06340b9cf175a87ecfa81775dffcc7d88a8abb0c
+  data.tar.gz: 92ddc3cd73f7af70f25e735fd8ac63dcbca6ed48fd78ba3510be0171ea180d006f399ff60e215bdf1b9159e52b721916c59fcb7278c322fb0ffe789db5900602

data/CHANGELOG.md

@@ -17,6 +17,59 @@
 
 *No unreleased changes yet. Check back after the next release.*
 
+## v15.5.0
+
+Changes since `v15.4.1`.
+
+This release makes the exec-backed tools report how long a command ran, calls a
+tracer's `on_exit` hook once when the last scope ends, and resolves a `Proc`
+parameter type when a schema is serialized. It also fixes `LLM::Schema.to_s`
+for a schema with a deferred type.
+
+### Tools
+
+* **tools: report how long a command ran** <br>
+  [`LLM::Tool::Exec#call`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool/Exec.html#call-instance_method)
+  now includes a `duration` field in its result, a string such as `"0.4 seconds"`
+  that reports how long the command took. The tools that route through `exec`
+  (`git`, `rg`, `mkdir`, `ruby`, and `bundle`) return it as well. A command that
+  was not found still returns the error hash, which has no `duration`.
+
+### Tracers
+
+* **tracer: call `on_exit` once, when the last scope ends** <br>
+  [`LLM::Tracer#on_exit`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html#on_exit-instance_method)
+  is a new hook, called once when the last scope that is open for a tracer
+  ends. A tracer that holds a connection, a file, or a span releases it here.
+  The count of open scopes belongs to the tracer rather than to the thread that
+  opened one, because a tool runs on a thread of its own and scopes the turn's
+  tracer while it does. Before this, a tool's scope looked like the outermost
+  one: a tracer that released its resource in `on_exit` lost it in the middle
+  of the turn, and the rest of the trace was written without it.
+
+### Schema
+
+* **schema: resolve a parameter type when the schema is serialized** <br>
+  [`LLM::Schema`](https://r.uby.dev/api-docs/llm.rb/LLM/Schema.html) and
+  [`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html) accept a `Proc`
+  as the type of a parameter or a property, and the proc is called when the
+  schema is serialized rather than when the class that declares it is defined.
+  A type that is only known at runtime no longer has to be known at load time:
+  `parameter :kind, proc { Enum[*Article.kinds] }, "The article kind"` asks the
+  model for the values a store holds right now, and the proc runs again on the
+  next request, so it sees the values that are current then. Whatever the proc
+  returns is resolved in turn, so it may be a leaf, a class, or an array of
+  either, and the description, `required`, `default`, and `enum` given with the
+  parameter are applied to it.
+
+* **schema: keep `to_s` working for a deferred type** <br>
+  Fix a bug where a schema with a proc-typed property raised from
+  [`LLM::Schema.to_s`](https://r.uby.dev/api-docs/llm.rb/LLM/Schema.html#to_s-class_method)
+  (and so from `inspect` and `p`) because the placeholder resolved its leaf
+  through a private method that shadowed the one `LLM::Schema::Leaf#default`
+  uses. Serialization, and therefore the request sent to a provider, was
+  unaffected.
+
 ## v15.4.1
 
 Changes since `v15.4.0`.

data/README.md

@@ -1092,9 +1092,9 @@ web</a>.
 
 The llm.rb project was started more than three
 years ago by
-[@0x1eef](https://github.com/0x1eef) and
-[@antaz](https://github.com/0x1eef). The primary
-maintainer is [@0x1eef](https://github.com/0x1eef).
+[@altruby](https://github.com/altruby) and
+[@antaz](https://github.com/altruby). The primary
+maintainer is [@altruby](https://github.com/altruby).
 Over those three years multiple other contributors have
 contributed to llm.rb as well, and new contributors are
 always welcome.

data/docs/deepdive/reference/tracer.md

@@ -156,6 +156,16 @@ Three more hooks cover a local tool call. `on_tool_start` fires before
 the tool runs and returns the span that `on_tool_finish` and
 `on_tool_error` receive.
 
+A tracer's own lifetime is bracketed as well. `on_exit` fires once,
+when the last scope that is open for that tracer ends. That scope can
+belong to a different thread than the one that opened the first: a tool
+runs on a thread of its own and scopes the turn's tracer while it does.
+`on_exit` is called after the scoped lookup has been restored, so a
+tracer that asks its provider for the current tracer from inside it
+sees the tracer the next request will see. A tracer can be scoped again
+afterwards, so it has to remain usable after `on_exit`, and `on_exit`
+may be called more than once over its life.
+
 A turn is additionally bracketed with `start_trace` and `stop_trace`.
 The runtime calls them around every agent turn with a `trace_group_id`,
 and a tracer that supports it (such as

data/lib/llm/function.rb

@@ -199,6 +199,12 @@ class LLM::Function
 
   ##
   # Set (or get) the function parameters
+  #
+  # @note
+  #  A parameter type that is given as a proc is resolved when the
+  #  parameters are rendered rather than here, so a tool can say
+  #  `parameter :fruit, proc { Enum[Fruit.keys] }` and ask the model
+  #  for what is available at the time of the call.
   # @yieldparam [LLM::Schema] schema The schema object
   # @return [LLM::Schema::Leaf, nil]
   def params

data/lib/llm/provider.rb

@@ -396,9 +396,18 @@ class LLM::Provider
   # Set the provider's default tracer
   # This tracer is shared by the provider instance and becomes the fallback
   # whenever no scoped override is active.
+  #
+  # A tracer assigned this way is not scoped, so nothing declares when it
+  # is finished and {LLM::Tracer#on_exit} never fires for it. Release what
+  # it holds when the provider is done with, or scope it with
+  # {#with_tracer} instead.
+  #
   # @example
   #   llm = LLM.openai(key: ENV["KEY"])
   #   llm.tracer = LLM::Tracer.logger(llm, path: "/path/to/log.txt")
+  #   llm.with_tracer(llm.tracer) do
+  #     # ...
+  #   end
   # @param [LLM::Tracer] tracer
   #  A tracer
   # @return [void]
@@ -418,19 +427,25 @@ class LLM::Provider
   # @yield
   # @return [Object]
   def with_tracer(tracer)
+    scoped = tracer || LLM::Tracer::Null.new(self)
     wm = weakmaps.tracer
     had_override = wm.key?(self)
     previous = wm[self]
-    wm[self] = tracer || LLM::Tracer::Null.new(self)
+    wm[self] = scoped
+    entered = LLM::Tracer.enter(scoped)
     yield
   ensure
     if had_override
       wm[self] = previous
-    elsif wm.respond_to?(:delete)
-      wm.delete(self)
     else
-      wm[self] = nil
+      wm.respond_to?(:delete) ? wm.delete(self) : wm[self] = nil
     end
+    ##
+    # The scope this tracer was opened for is over, and the tracer is
+    # told so when the last one that is open for it ends - on whichever
+    # thread that happens, because a tool opens a scope of its own for
+    # the turn's tracer.
+    LLM::Tracer.exit(scoped) if entered
   end
 
   ##

data/lib/llm/schema/deferred.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+class LLM::Schema
+  ##
+  # {LLM::Schema::Deferred LLM::Schema::Deferred} is a placeholder for a
+  # leaf that is resolved when the schema is rendered.
+  #
+  # A parameter type that can only be known at runtime - an enum of the
+  # keys a store holds right now, for example - is given as a proc. The
+  # proc is called when the schema is rendered rather than when the class
+  # that declares the parameter is defined, so declaring a tool does not
+  # touch whatever the type depends on:
+  #
+  #   parameter :fruit, proc { Enum[*Fruit.kinds]}, "A fruit", required: true
+  #
+  # Whatever the proc returns is resolved in turn, so it can return a
+  # leaf, a class, or an array of either.
+  #
+  # @api private
+  class Deferred < Leaf
+    ##
+    # @param [Proc] block
+    #  A block that returns the leaf to resolve to
+    # @return [LLM::Schema::Deferred]
+    def initialize(&block)
+      super()
+      @block = block
+    end
+
+    ##
+    # @return [Hash]
+    def to_h
+      resolved_leaf.to_h
+    end
+
+    ##
+    # @param [Hash] options
+    # @return [String]
+    def to_json(options = {})
+      resolved_leaf.to_h.to_json(options)
+    end
+
+    ##
+    # @return [String]
+    def to_s
+      resolved_leaf.to_s
+    end
+
+    private
+
+    ##
+    # Returns the leaf the block resolves to, with the settings that
+    # were given to the placeholder applied to it.
+    #
+    # The settings live on the placeholder, because they are given
+    # before the leaf it stands for exists.
+    # @return [LLM::Schema::Leaf]
+    def resolved_leaf
+      leaf = @block.call
+      leaf.description(@description) if @description
+      leaf.default(@default) unless @default.nil?
+      leaf.enum(*@enum) if @enum
+      leaf.const(@const) unless @const.nil?
+      leaf.required if required?
+      leaf
+    end
+  end
+end

data/lib/llm/schema/leaf.rb

@@ -12,6 +12,9 @@ class LLM::Schema
     # @return [Integer, nil]
     attr_accessor :index
 
+    ##
+    # Returns a leaf.
+    # @return [LLM::Schema::Leaf]
     def initialize
       @description = nil
       @default = nil

data/lib/llm/schema.rb

@@ -36,6 +36,7 @@ class LLM::Schema
   require_relative "schema/parser"
   require_relative "schema/renderer"
   require_relative "schema/leaf"
+  require_relative "schema/deferred"
   require_relative "schema/object"
   require_relative "schema/array"
   require_relative "schema/all_of"
@@ -56,9 +57,21 @@ class LLM::Schema
   module Utils
     extend self
 
+    ##
+    # Resolves a parameter type into a leaf of the schema.
+    #
+    # A proc is called for its type, and whatever it returns is resolved
+    # in turn, so a type that can only be known at runtime - an enum of
+    # the keys a store holds right now - can be declared without being
+    # evaluated when the class that declares it is defined.
+    # @param [LLM::Schema] schema
+    # @param [Object] type
+    # @return [LLM::Schema::Leaf]
     def resolve(schema, type)
       if LLM::Schema::Leaf === type
         type
+      elsif ::Proc === type
+        Deferred.new { resolve(schema, type.call) }
       elsif ::Array === type
         resolve_array(schema, type)
       elsif Class === type && type.respond_to?(:object)

data/lib/llm/tool/param.rb

@@ -22,8 +22,10 @@ class LLM::Tool
     ##
     # @param name [Symbol]
     #   The name of a parameter
-    # @param type [LLM::Schema::Leaf, Class]
-    #   The parameter type (eg String)
+    # @param type [LLM::Schema::Leaf, Class, Proc]
+    #   The parameter type (eg String). A proc is called for the type
+    #   when the parameters are read, so a type that can only be known
+    #   at runtime does not have to be known when the class is defined.
     # @param description [String]
     #   The description of a property
     # @param options [Hash]

data/lib/llm/tools/exec.rb

@@ -54,7 +54,8 @@ class LLM::Tool
     #  the max number of bytes to emit
     # @return [Hash]
     def call(arguments: [], timeout: 60, max_bytes: self.class.max_bytes)
-      name = arguments[0]
+      startat = now
+      name    = arguments[0]
       command = spawn(name:, arguments: arguments[1..], env:, max_bytes:)
       wait(command:, timeout:)
       if command.not_found?
@@ -62,7 +63,8 @@ class LLM::Tool
       else
         {ok: command.success?,
         stdout: truncate(command.stdout, max_bytes:),
-        stderr: truncate(command.stderr, max_bytes:)}
+        stderr: truncate(command.stderr, max_bytes:),
+        duration: "#{(now - startat).round(1)} seconds"}
       end
     rescue LLM::Interrupt
       command.kill! if command&.running?
@@ -72,5 +74,11 @@ class LLM::Tool
     private
 
     attr_reader :env
+
+    ##
+    # @return [Float]
+    def now
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
   end
 end

data/lib/llm/tracer/registry.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+class LLM::Tracer
+  ##
+  # {LLM::Tracer::Registry LLM::Tracer::Registry} counts the scopes that
+  # are open for each tracer.
+  #
+  # The count belongs to the tracer, not to the thread that opened a
+  # scope. A tool runs on a thread of its own and opens a scope there for
+  # the turn's tracer, and a count kept per thread would let that scope
+  # look like the outermost one: it would end the tracer while the turn
+  # is still running, and every write after it would fail.
+  #
+  # @api private
+  class Registry
+    ##
+    # @return [LLM::Tracer::Registry]
+    def initialize
+      @counts = ObjectSpace::WeakMap.new
+      @mutex  = Mutex.new
+    end
+
+    ##
+    # Records that a scope was opened for a tracer.
+    # @param [LLM::Tracer] tracer
+    # @return [LLM::Tracer::Registry]
+    #  Returns self, so a caller can tell the scope was recorded
+    def enter(tracer)
+      @mutex.synchronize do
+        @counts[tracer] = count(tracer) + 1
+      end
+      self
+    end
+
+    ##
+    # Records that a scope was closed, and tells the tracer it is
+    # finished when it was the last one that was open.
+    #
+    # The count is brought up to date under the lock, and the tracer is
+    # told after the lock is released: a tracer that calls back into the
+    # provider from `on_exit` would otherwise deadlock against itself.
+    #
+    # An `exit` without a matching `enter` is ignored: no scope is open,
+    # so there is nothing to end, and the tracer is not told it is
+    # finished when no scope was holding it.
+    # @param [LLM::Tracer] tracer
+    # @return [void]
+    def exit(tracer)
+      last = @mutex.synchronize do
+        open = count(tracer)
+        if open.zero?
+          false
+        elsif open == 1
+          forget(tracer)
+          true
+        else
+          @counts[tracer] = open - 1
+          false
+        end
+      end
+      tracer.on_exit if last
+    end
+
+    private
+
+    ##
+    # Drops a tracer that has no open scopes left.
+    # @param [LLM::Tracer] tracer
+    # @return [Boolean]
+    def forget(tracer)
+      if @counts.respond_to?(:delete)
+        @counts.delete(tracer)
+      else
+        @counts[tracer] = 0
+      end
+      true
+    end
+
+    ##
+    # @param [LLM::Tracer] tracer
+    # @return [Integer]
+    def count(tracer)
+      @counts[tracer] || 0
+    end
+  end
+end

data/lib/llm/tracer.rb

@@ -13,6 +13,39 @@ module LLM
     require_relative "tracer/telemetry"
     require_relative "tracer/null"
     require_relative "tracer/pretty_logger"
+    require_relative "tracer/registry"
+
+    ##
+    # Returns the registry that counts the open scopes for each tracer.
+    # @api private
+    # @return [LLM::Tracer::Registry]
+    def self.registry
+      LLM.lock(:registry) do
+        @registry ||= Registry.new
+      end
+    end
+
+    ##
+    # Records that a scope was opened for a tracer.
+    # @api private
+    # @see LLM::Tracer::Registry#enter
+    # @param [LLM::Tracer] tracer
+    # @return [LLM::Tracer::Registry]
+    #  Returns the registry, so a caller can tell the scope was recorded
+    def self.enter(tracer)
+      registry.enter(tracer)
+    end
+
+    ##
+    # Records that a scope was closed for a tracer, and tells the tracer
+    # it is finished when it was the last one that was open.
+    # @api private
+    # @see LLM::Tracer::Registry#exit
+    # @param [LLM::Tracer] tracer
+    # @return [void]
+    def self.exit(tracer)
+      registry.exit(tracer)
+    end
 
     ##
     # Builds a {LLM::Tracer::PrettyLogger} for a provider.
@@ -61,6 +94,29 @@ module LLM
       @options = {}
     end
 
+    ##
+    # Called when the tracer is no longer in use.
+    #
+    # A tracer that holds something - a file, a span, a connection pool -
+    # releases it here. The default does nothing, because most tracers hold
+    # nothing that needs releasing.
+    #
+    # It is called once, when the last scope that is open for this tracer
+    # ends, and that scope can have been opened by another thread: a tool
+    # runs on a thread of its own and scopes the turn's tracer while it
+    # does. It is called after the scoped lookup is restored, so a tracer
+    # that reads {LLM::Provider#tracer} inside it sees the tracer that the
+    # next request will see.
+    #
+    # A tracer can be scoped again afterwards, on a later turn, so it has
+    # to stay usable after this call, and it has to tolerate the call
+    # happening more than once over its life.
+    # @see LLM::Tracer::Registry
+    # @see LLM::Provider#with_tracer
+    # @return [void]
+    def on_exit
+    end
+
     ##
     # Called before an LLM provider request is executed.
     # @param [String] operation

data/lib/llm/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module LLM
-  VERSION = "15.4.1"
+  VERSION = "15.5.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: llm.rb
 version: !ruby/object:Gem::Version
-  version: 15.4.1
+  version: 15.5.0
 platform: ruby
 authors:
 - Robert Gleeson
@@ -624,6 +624,7 @@ files:
 - lib/llm/schema/any_of.rb
 - lib/llm/schema/array.rb
 - lib/llm/schema/boolean.rb
+- lib/llm/schema/deferred.rb
 - lib/llm/schema/enum.rb
 - lib/llm/schema/integer.rb
 - lib/llm/schema/leaf.rb
@@ -665,6 +666,7 @@ files:
 - lib/llm/tracer/logger.rb
 - lib/llm/tracer/null.rb
 - lib/llm/tracer/pretty_logger.rb
+- lib/llm/tracer/registry.rb
 - lib/llm/tracer/telemetry.rb
 - lib/llm/transformer.rb
 - lib/llm/transformer/null.rb