llm.rb 0.14.2 → 0.16.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 41efcc6a142e5863b3b1d2ab1b1c8a452b5c8e8098bf48918c3bd5b16abfcdd2
-  data.tar.gz: d2bfdfd690070d6dbca8411874d2970e470b37ed67139976026120b2806e3421
+  metadata.gz: 38502ab4a41dba8177cb7b21db68f3e0dd5492323ac8b132b1775926b46ffffc
+  data.tar.gz: ebe196962c43934ae979e298f80b4bc2e30a147ad0f42595eef59880be9fc01e
 SHA512:
-  metadata.gz: 7a10131e21e424e4620b155ef4e8fd23617aecc5ee9141e6a6f0e09b99b042286992f6b27299502f0d7664a92c62e520bf99a7f8aa0836ba0973292c74ab5d80
-  data.tar.gz: 57e5110fc829a790ae064c5a52b2ae856fd5566b5256444148008dd9f3dc170b3cbf57fda0c09f7ecc5630215c0c7e236375d063943d9b791a034359569c7ecb
+  metadata.gz: 5100d71b851771137a86e799bc2cadab360fc0ced297288b09fa701a8f434c671fe739427b889243e9704c5ae2a05b6b8761c85f0b0bea9268700bf770e80f13
+  data.tar.gz: 3aed0f826c229a37d30b2f0d41678976ba00df72216588a9302320c52e22d9d1b814df0b76bca8f6e713e105d73006339121a8bb900a73ac8896cc1f7c3f0051

data/README.md

@@ -1,21 +1,21 @@
 ## About
 
 llm.rb is a zero-dependency Ruby toolkit for Large Language Models that
-includes OpenAI, Gemini, Anthropic, xAI (grok), DeepSeek, Ollama, and
+includes OpenAI, Gemini, Anthropic, xAI (Grok), DeepSeek, Ollama, and
 LlamaCpp. The toolkit includes full support for chat, streaming, tool calling,
 audio, images, files, and structured outputs (JSON Schema).
 
-The library provides a common, uniform interface for all the providers and
-features it supports, in addition to provider-specific features as well. Keep
-reading to find out more.
-
 ## Quick start
 
 #### Demo
 
+This cool demo writes a new [llm-shell](https://github.com/llmrb/llm-shell#readme) command
+with the help of [llm.rb](https://github.com/llmrb/llm#readme). <br> Similar-ish to
+GitHub Copilot but for the terminal.
+
 <details>
-  <summary>Play</summary>
-  <img src="share/llm-shell/examples/demo.gif/">
+  <summary>Start demo</summary>
+  <img src="https://github.com/llmrb/llm/blob/main/share/llm-shell/examples/demo.gif?raw=true" alt="llm-shell demo" />
 </details>
 
 #### Guides
@@ -26,11 +26,13 @@ reading to find out more.
   a blog post that implements an age estimation tool
 * [How to edit an image with Gemini](https://0x1eef.github.io/posts/how-to-edit-images-with-gemini/) &ndash;
   a blog post that implements image editing with Gemini
+* [Fast sailing with persistent connections](https://0x1eef.github.io/posts/persistent-connections-with-llm.rb/) &ndash;
+  a blog post that optimizes performance with a thread-safe connection pool
 
 #### Ecosystem
 
-* [llm-shell](https://github.com/llmrb/llm-shell) &ndash; is a developer-oriented console for Large Language Model communication
-* [llm-spell](https://github.com/llmrb/llm-spell) &ndash; is a utility that can correct spelling mistakes with a Large Language Model
+* [llm-shell](https://github.com/llmrb/llm-shell) &ndash; a developer-oriented console for Large Language Model communication
+* [llm-spell](https://github.com/llmrb/llm-spell) &ndash; a utility that can correct spelling mistakes with a Large Language Model
 
 ## Features
 
@@ -38,6 +40,7 @@ reading to find out more.
 - ✅ A single unified interface for multiple providers
 - 📦 Zero dependencies outside Ruby's standard library
 - 🚀 Smart API design that minimizes the number of requests made
+- ♻️ Optional: per-provider, process-wide connection pool via net-http-persistent
 
 #### Chat, Agents
 - 🧠 Stateless and stateful chat via completions and responses API
@@ -114,13 +117,30 @@ llm = LLM.ollama(key: nil)
 llm = LLM.llamacpp(key: nil)
 ```
 
+#### Persistence
+
+The llm.rb library can maintain a process-wide connection pool
+for each provider that is instantiated. This feature can improve
+performance but it is optional, the implementation depends on
+[net-http-persistent](https://github.com/dbrain/net-http-persistent),
+and the gem should be installed separately:
+
+```ruby
+#!/usr/bin/env ruby
+require "llm"
+
+llm = LLM.openai(key: ENV["KEY"], persistent: true)
+res = llm.responses.create "Hello world"
+llm.responses.create "Adios", last_response_id: res.response_id
+```
+
 ### Conversations
 
 #### Completions
 
 > This example uses the stateless chat completions API that all
 > providers support. A similar example for OpenAI's stateful
-> responses API is available in the [docs/](docs/OPENAI.md#responses)
+> responses API is available in the [docs/](https://0x1eef.github.io/x/llm.rb/file.OPENAI.html#responses)
 > directory.
 
 The following example creates an instance of
@@ -138,15 +158,14 @@ require "llm"
 llm  = LLM.openai(key: ENV["KEY"])
 bot  = LLM::Bot.new(llm)
 url  = "https://en.wikipedia.org/wiki/Special:FilePath/Cognac_glass.jpg"
-msgs = bot.chat do |prompt|
-  prompt.system "Your task is to answer all user queries"
-  prompt.user ["Tell me about this URL", URI(url)]
-  prompt.user ["Tell me about this PDF", File.open("handbook.pdf", "rb")]
-  prompt.user "Are the URL and PDF similar to each other?"
-end
+
+bot.chat "Your task is to answer all user queries", role: :system
+bot.chat ["Tell me about this URL", URI(url)], role: :user
+bot.chat ["Tell me about this PDF", File.open("handbook.pdf", "rb")], role: :user
+bot.chat "Are the URL and PDF similar to each other?", role: :user
 
 # At this point, we execute a single request
-msgs.each { print "[#{_1.role}] ", _1.content, "\n" }
+bot.messages.each { print "[#{_1.role}] ", _1.content, "\n" }
 ```
 
 #### Streaming
@@ -154,7 +173,8 @@ msgs.each { print "[#{_1.role}] ", _1.content, "\n" }
 > There Is More Than One Way To Do It (TIMTOWTDI) when you are
 > using llm.rb &ndash; and this is especially true when it
 > comes to streaming. See the streaming documentation in
-> [docs/](docs/STREAMING.md#scopes) for more details.
+> [docs/](https://0x1eef.github.io/x/llm.rb/file.STREAMING.html#scopes)
+> for more details.
 
 The following example streams the messages in a conversation
 as they are generated in real-time. The `stream` option can
@@ -268,6 +288,43 @@ bot.chat bot.functions.map(&:call) # report return value to the LLM
 # {stderr: "", stdout: "FreeBSD"}
 ```
 
+#### Provider
+
+The
+[LLM::Function](https://0x1eef.github.io/x/llm.rb/LLM/Function.html)
+class can define a local function that can be called by a provider on your behalf,
+and the
+[LLM::Tool](https://0x1eef.github.io/x/llm.rb/LLM/Tool.html)
+class represents a tool that is defined and implemented by a provider, and we can
+request that the provider call the tool on our behalf. That's the primary difference
+between a function implemented locally and a tool implemented by a provider. The
+available tools depend on the provider, and the following example uses the
+OpenAI provider to execute Python code on OpenAI's servers:
+
+```ruby
+#!/usr/bin/env ruby
+require "llm"
+
+llm = LLM.openai(key: ENV["KEY"])
+res = llm.responses.create "Run: 'print(\"hello world\")'", tools: [llm.tool(:code_interpreter)]
+print res.output_text, "\n"
+```
+
+#### Web Search
+
+A common tool among all providers is the ability to perform a web search, and
+the following example uses the OpenAI provider to search the web using the
+Web Search tool. This can also be done with the Anthropic and Gemini providers:
+
+```ruby
+#!/usr/bin/env ruby
+require "llm"
+
+llm = LLM.openai(key: ENV["KEY"])
+res = llm.web_search(query: "summarize today's news")
+print res.output_text, "\n"
+```
+
 ### Files
 
 #### Create
@@ -509,6 +566,23 @@ bot.chat "Hello #{model.id} :)"
 bot.messages.select(&:assistant?).each { print "[#{_1.role}] ", _1.content, "\n" }
 ```
 
+## Reviews
+
+I supplied both Gemini and DeepSeek with the contents of [lib/](https://github.com/llmrb/llm/tree/main/lib)
+and [README.md](https://github.com/llmrb/llm#readme) via [llm-shell](https://github.com/llmrb/llm-shell#readme).
+Their feedback was way more positive than I could have imagined 😅 These are genuine responses though, with no
+special prompting or engineering. I just provided them with the source code and asked for their opinion.
+
+<details>
+  <summary>Review by Gemini</summary>
+  <img src="https://github.com/llmrb/llm/blob/main/share/llm-shell/examples/gemini.png?raw=true" alt="Gemini review" />
+</details>
+
+<details>
+  <summary>Review by DeepSeek</summary>
+  <img src="https://github.com/llmrb/llm/blob/main/share/llm-shell/examples/deepseek.png?raw=true" alt="DeepSeek review" />
+</details>
+
 ## Documentation
 
 ### API
@@ -528,12 +602,6 @@ over or doesn't cover at all. The API reference is available at
   a blog post that implements image editing with Gemini
 * [docs/](docs/) &ndash; the docs directory contains additional guides
 
-
-## See also
-
-* [llm-shell](https://github.com/llmrb/llm-shell) &ndash; a shell that uses llm.rb to
-  provide a command-line interface to LLMs.
-
 ## Install
 
 llm.rb can be installed via rubygems.org:

data/lib/llm/buffer.rb

@@ -135,10 +135,10 @@ module LLM
       params = [
         *oldparams,
         params.merge(input: messages),
-        @response ? {previous_response_id: @response.id} : {}
+        @response ? {previous_response_id: @response.response_id} : {}
       ].inject({}, &:merge!)
       @response = @provider.responses.create(message.content, params.merge(role:))
-      @completed.concat([*pendings, message, *@response.outputs[0]])
+      @completed.concat([*pendings, message, *@response.choices[0]])
       @pending.clear
     end
   end

data/lib/llm/client.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module LLM
+  ##
+  # @api private
+  module Client
+    private
+
+    ##
+    # @api private
+    def persistent_client
+      mutex.synchronize do
+        if clients[client_id]
+          clients[client_id]
+        else
+          require "net/http/persistent" unless defined?(Net::HTTP::Persistent)
+          client = Net::HTTP::Persistent.new(name: self.class.name)
+          client.read_timeout = timeout
+          clients[client_id] = client
+        end
+      end
+    end
+
+    ##
+    # @api private
+    def transient_client
+      client = Net::HTTP.new(host, port)
+      client.read_timeout = timeout
+      client.use_ssl = ssl
+      client
+    end
+
+    def client_id = "#{host}:#{port}:#{timeout}:#{ssl}"
+    def clients = self.class.clients
+    def mutex = self.class.mutex
+  end
+end

data/lib/llm/function.rb

@@ -30,7 +30,7 @@
 #     fn.register(System)
 #   end
 class LLM::Function
-  class Return < Struct.new(:id, :value)
+  class Return < Struct.new(:id, :name, :value)
   end
 
   ##
@@ -92,7 +92,7 @@ class LLM::Function
   # @return [LLM::Function::Return] The result of the function call
   def call
     runner = ((Class === @runner) ? @runner.new : @runner)
-    Return.new(id, runner.call(**arguments))
+    Return.new(id, name, runner.call(**arguments))
   ensure
     @called = true
   end
@@ -106,7 +106,7 @@ class LLM::Function
   #   bot.chat bot.functions.map(&:cancel)
   # @return [LLM::Function::Return]
   def cancel(reason: "function call cancelled")
-    Return.new(id, {cancelled: true, reason:})
+    Return.new(id, name, {cancelled: true, reason:})
   ensure
     @cancelled = true
   end

data/lib/llm/message.rb

@@ -127,6 +127,35 @@ module LLM
       extra[:response]
     end
 
+    ##
+    # @note
+    #  This method might return annotations for assistant messages,
+    #  and it returns an empty array for non-assistant messages
+    # Returns annotations associated with the message
+    # @return [Array<LLM::Object>]
+    def annotations
+      @annotations ||= LLM::Object.from_hash(extra["annotations"] || [])
+    end
+
+    ##
+    # @note
+    #  This method returns token usage for assistant messages,
+    #  and it returns an empty object for non-assistant messages
+    # Returns token usage statistics
+    # @return [LLM::Object]
+    def usage
+      @usage ||= if response
+        LLM::Object.from_hash({
+          input_tokens: response.prompt_tokens || 0,
+          output_tokens: response.completion_tokens || 0,
+          total_tokens: response.total_tokens || 0
+        })
+      else
+        LLM::Object.from_hash({})
+      end
+    end
+    alias_method :token_usage, :usage
+
     ##
     # Returns a string representation of the message
     # @return [String]

data/lib/llm/object.rb

@@ -69,6 +69,12 @@ class LLM::Object < BasicObject
     to_h.dig(...)
   end
 
+  ##
+  # @return [Hash]
+  def slice(...)
+    to_h.slice(...)
+  end
+
   private
 
   def method_missing(m, *args, &b)

data/lib/llm/provider.rb

@@ -7,6 +7,19 @@
 # @abstract
 class LLM::Provider
   require "net/http"
+  require_relative "client"
+  include LLM::Client
+
+  @@clients = {}
+  @@mutex = Mutex.new
+
+  ##
+  # @api private
+  def self.clients = @@clients
+
+  ##
+  # @api private
+  def self.mutex = @@mutex
 
   ##
   # @param [String, nil] key
@@ -19,11 +32,17 @@ class LLM::Provider
   #  The number of seconds to wait for a response
   # @param [Boolean] ssl
   #  Whether to use SSL for the connection
-  def initialize(key:, host:, port: 443, timeout: 60, ssl: true)
+  # @param [Boolean] persistent
+  #  Whether to use a persistent connection.
+  #  Requires the net-http-persistent gem.
+  def initialize(key:, host:, port: 443, timeout: 60, ssl: true, persistent: false)
     @key = key
-    @client = Net::HTTP.new(host, port)
-    @client.use_ssl = ssl
-    @client.read_timeout = timeout
+    @host = host
+    @port = port
+    @timeout = timeout
+    @ssl = ssl
+    @client = persistent ? persistent_client : transient_client
+    @base_uri = URI("#{ssl ? "https" : "http"}://#{host}:#{port}/")
   end
 
   ##
@@ -217,9 +236,46 @@ class LLM::Provider
     tap { (@headers ||= {}).merge!(headers) }
   end
 
+  ##
+  # @note
+  #  This method might be outdated, and the {LLM::Provider#tool LLM::Provider#tool}
+  #  method can be used if a tool is not found here.
+  # Returns all known tools provided by a provider.
+  # @return [String => LLM::Tool]
+  def tools
+    {}
+  end
+
+  ##
+  # @note
+  #   OpenAI, Anthropic, and Gemini provide platform-tools for things
+  #   like web search, and more.
+  # Returns a tool provided by a provider.
+  # @example
+  #   llm   = LLM.openai(key: ENV["KEY"])
+  #   tools = [llm.tool(:web_search)]
+  #   res   = llm.responses.create("Summarize today's news", tools:)
+  #   print res.output_text, "\n"
+  # @param [String, Symbol] name The name of the tool
+  # @param [Hash] options Configuration options for the tool
+  # @return [LLM::Tool]
+  def tool(name, options = {})
+    LLM::Tool.new(name, options, self)
+  end
+
+  ##
+  # Provides a web search capability
+  # @param [String] query The search query
+  # @raise [NotImplementedError]
+  #  When the method is not implemented by a subclass
+  # @return [LLM::Response]
+  def web_search(query:)
+    raise NotImplementedError
+  end
+
   private
 
-  attr_reader :client
+  attr_reader :client, :base_uri, :host, :port, :timeout, :ssl
 
   ##
   # The headers to include with a request
@@ -268,9 +324,10 @@ class LLM::Provider
   # @raise [SystemCallError]
   #  When there is a network error at the operating system level
   # @return [Net::HTTPResponse]
-  def execute(request:, stream: nil, &b)
+  def execute(request:, stream: nil, stream_parser: self.stream_parser, &b)
+    args = (Net::HTTP === client) ? [request] : [URI.join(base_uri, request.path), request]
     res = if stream
-      client.request(request) do |res|
+      client.request(*args) do |res|
         handler = event_handler.new stream_parser.new(stream)
         parser = LLM::EventStream::Parser.new
         parser.register(handler)
@@ -284,8 +341,8 @@ class LLM::Provider
         parser&.free
       end
     else
-      b ? client.request(request) { (Net::HTTPSuccess === _1) ? b.call(_1) : _1 } :
-          client.request(request)
+      b ? client.request(*args) { (Net::HTTPSuccess === _1) ? b.call(_1) : _1 } :
+          client.request(*args)
     end
     handle_response(res)
   end

data/lib/llm/providers/anthropic/format.rb

@@ -24,7 +24,7 @@ class LLM::Anthropic
     def format_tools(params)
       return {} unless params and params[:tools]&.any?
       tools = params[:tools]
-      {tools: tools.map { _1.format(self) }}
+      {tools: tools.map { _1.respond_to?(:format) ? _1.format(self) : _1 }}
     end
   end
 end

data/lib/llm/providers/anthropic/response/web_search.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module LLM::Anthropic::Response
+  ##
+  # The {LLM::Anthropic::Response::WebSearch LLM::Anthropic::Response::WebSearch}
+  # module provides methods for accessing web search results from a web search
+  # tool call made via the {LLM::Provider#web_search LLM::Provider#web_search}
+  # method.
+  module WebSearch
+    ##
+    # Returns one or more search results
+    # @return [Array<LLM::Object>]
+    def search_results
+      LLM::Object.from_hash(
+        content
+          .select { _1["type"] == "web_search_tool_result" }
+          .flat_map { |n| n.content.map { _1.slice(:title, :url) } }
+      )
+    end
+  end
+end

data/lib/llm/providers/anthropic.rb

@@ -15,6 +15,7 @@ module LLM
   #   bot.messages.select(&:assistant?).each { print "[#{_1.role}]", _1.content, "\n" }
   class Anthropic < Provider
     require_relative "anthropic/response/completion"
+    require_relative "anthropic/response/web_search"
     require_relative "anthropic/format"
     require_relative "anthropic/error_handler"
     require_relative "anthropic/stream_parser"
@@ -83,6 +84,35 @@ module LLM
       "claude-sonnet-4-20250514"
     end
 
+    ##
+    # @note
+    #  This method includes certain tools that require configuration
+    #  through a set of options that are easier to set through the
+    #  {LLM::Provider#tool LLM::Provider#tool} method.
+    # @see https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/web-search-tool Anthropic docs
+    # @return (see LLM::Provider#tools)
+    def tools
+      {
+        bash: tool(:bash, type: "bash_20250124"),
+        web_search: tool(:web_search, type: "web_search_20250305", max_uses: 5),
+        text_editor: tool(:str_replace_based_edit_tool, type: "text_editor_20250728", max_characters: 10_000)
+      }
+    end
+
+    ##
+    # A convenience method for performing a web search using the
+    # Anthropic web search tool.
+    # @example
+    #   llm = LLM.anthropic(key: ENV["KEY"])
+    #   res = llm.web_search(query: "summarize today's news")
+    #   res.search_results.each { |item| print item.title, ": ", item.url, "\n" }
+    # @param query [String] The search query.
+    # @return [LLM::Response] The response from the LLM provider.
+    def web_search(query:)
+      complete(query, tools: [tools[:web_search]])
+        .extend(LLM::Anthropic::Response::WebSearch)
+    end
+
     private
 
     def headers

data/lib/llm/providers/gemini/format/completion_format.rb

@@ -43,7 +43,7 @@ module LLM::Gemini::Format
       when LLM::Message
         format_content(content.content)
       when LLM::Function::Return
-        [{text: JSON.dump(content.value)}]
+        [{functionResponse: {name: content.name, response: content.value}}]
       else
         prompt_error!(content)
       end

data/lib/llm/providers/gemini/format.rb

@@ -32,8 +32,9 @@ class LLM::Gemini
     # @return [Hash]
     def format_tools(params)
       return {} unless params and params[:tools]&.any?
-      functions = params.delete(:tools).grep(LLM::Function)
-      {tools: {functionDeclarations: functions.map { _1.format(self) }}}
+      tools = params.delete(:tools)
+      platform, functions = [tools.grep(LLM::Tool), tools.grep(LLM::Function)]
+      {tools: [*platform, {functionDeclarations: functions.map { _1.format(self) }}]}
     end
   end
 end

data/lib/llm/providers/gemini/images.rb

@@ -44,7 +44,7 @@ class LLM::Gemini
     def create(prompt:, model: "gemini-2.0-flash-exp-image-generation", **params)
       req  = Net::HTTP::Post.new("/v1beta/models/#{model}:generateContent?key=#{key}", headers)
       body = JSON.dump({
-        contents: [{parts: [{text: system_prompt}, {text: prompt}]}],
+        contents: [{parts: [{text: create_prompt}, {text: prompt}]}],
         generationConfig: {responseModalities: ["TEXT", "IMAGE"]}
       }.merge!(params))
       req.body = body
@@ -69,7 +69,7 @@ class LLM::Gemini
       req   = Net::HTTP::Post.new("/v1beta/models/#{model}:generateContent?key=#{key}", headers)
       image = LLM.File(image)
       body  = JSON.dump({
-        contents: [{parts: [{text: prompt}, format.format_content(image)]}],
+        contents: [{parts: [{text: edit_prompt}, {text: prompt}, format.format_content(image)]}],
         generationConfig: {responseModalities: ["TEXT", "IMAGE"]}
       }.merge!(params)).b
       set_body_stream(req, StringIO.new(body))
@@ -94,12 +94,28 @@ class LLM::Gemini
       @provider.instance_variable_get(:@key)
     end
 
-    def system_prompt
+    def create_prompt
       <<~PROMPT
-        Your task is to generate one or more image(s) from
-        text I will provide to you. Your response *MUST* include
-        at least one image, and your response *MUST NOT* include
-        any text or other content.
+        ## Context
+        Your task is to generate one or more image(s) based on the user's instructions.
+        The user will provide you with text only.
+
+        ## Instructions
+        1. The model *MUST* generate image(s) based on the user text alone.
+        2. The model *MUST NOT* generate anything else.
+      PROMPT
+    end
+
+    def edit_prompt
+      <<~PROMPT
+        ## Context
+        Your task is to edit the provided image based on the user's instructions.
+        The user will provide you with both text and an image.
+
+        ## Instructions
+        1. The model *MUST* edit the provided image based on the user's instructions
+        2. The model *MUST NOT* generate a new image.
+        3. The model *MUST NOT* generate anything else.
       PROMPT
     end
 

data/lib/llm/providers/gemini/response/completion.rb

@@ -13,8 +13,9 @@ module LLM::Gemini::Response
     def format_choices
       candidates.map.with_index do |choice, index|
         choice = LLM::Object.from_hash(choice)
-        content = choice.content
-        role, parts = content.role, content.parts
+        content = choice.content || LLM::Object.new
+        role = content.role || "model"
+        parts = content.parts || [{"text" => choice.finishReason}]
         text  = parts.filter_map { _1["text"] }.join
         tools = parts.filter_map { _1["functionCall"] }
         extra = {index:, response: self, tool_calls: format_tool_calls(tools), original_tool_calls: tools}

data/lib/llm/providers/gemini/response/web_search.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module LLM::Gemini::Response
+  ##
+  # The {LLM::Gemini::Response::WebSearch LLM::Gemini::Response::WebSearch}
+  # module provides methods for accessing web search results from a web search
+  # tool call made via the {LLM::Provider#web_search LLM::Provider#web_search}
+  # method.
+  module WebSearch
+    ##
+    # Returns one or more search results
+    # @return [Array<LLM::Object>]
+    def search_results
+      LLM::Object.from_hash(
+        candidates[0]
+          .groundingMetadata
+          .groundingChunks
+          .map { {"url" => _1.web.uri, "title" => _1.web.title} }
+      )
+    end
+  end
+end

data/lib/llm/providers/gemini/stream_parser.rb

@@ -13,7 +13,7 @@ class LLM::Gemini
     # @param [#<<] io An IO-like object
     # @return [LLM::Gemini::StreamParser]
     def initialize(io)
-      @body = LLM::Object.new
+      @body = LLM::Object.from_hash({candidates: []})
       @io = io
     end
 
@@ -21,47 +21,64 @@ class LLM::Gemini
     # @param [Hash] chunk
     # @return [LLM::Gemini::StreamParser]
     def parse!(chunk)
-      tap { merge!(chunk) }
+      tap { merge_chunk!(LLM::Object.from_hash(chunk)) }
     end
 
     private
 
-    def merge!(chunk)
+    def merge_chunk!(chunk)
       chunk.each do |key, value|
-        if key == "candidates"
-          @body.candidates ||= []
+        if key.to_s == "candidates"
           merge_candidates!(value)
+        elsif key.to_s == "usageMetadata" &&
+            @body.usageMetadata.is_a?(LLM::Object) &&
+            value.is_a?(LLM::Object)
+          @body.usageMetadata = LLM::Object.from_hash(@body.usageMetadata.to_h.merge(value.to_h))
         else
           @body[key] = value
         end
       end
     end
 
-    def merge_candidates!(candidates)
-      candidates.each.with_index do |candidate, i|
-        if @body.candidates[i].nil?
-          merge_one(@body.candidates, candidate, i)
-        else
-          merge_two(@body.candidates, candidate, i)
+    def merge_candidates!(new_candidates_list)
+      new_candidates_list.each do |new_candidate_delta|
+        index = new_candidate_delta.index
+        @body.candidates[index] ||= LLM::Object.from_hash({content: {parts: []}})
+        existing_candidate = @body.candidates[index]
+        new_candidate_delta.each do |key, value|
+          if key.to_s == "content"
+            merge_candidate_content!(existing_candidate.content, value) if value
+          else
+            existing_candidate[key] = value # Overwrite other fields
+          end
         end
       end
     end
 
-    def merge_one(candidates, candidate, i)
-      candidate
-        .dig("content", "parts")
-        &.filter_map { _1["text"] }
-        &.each { @io << _1 if @io.respond_to?(:<<) }
-      candidates[i] = candidate
+    def merge_candidate_content!(existing_content, new_content_delta)
+      new_content_delta.each do |key, value|
+        if key.to_s == "parts"
+          existing_content.parts ||= []
+          merge_content_parts!(existing_content.parts, value) if value
+        else
+          existing_content[key] = value
+        end
+      end
     end
 
-    def merge_two(candidates, candidate, i)
-      parts = candidates[i].dig("content", "parts")
-      parts&.each&.with_index do |part, j|
-        if part["text"]
-          target = candidate["content"]["parts"][j]
-          part["text"] << target["text"]
-          @io << target["text"] if @io.respond_to?(:<<)
+    def merge_content_parts!(existing_parts, new_parts_delta)
+      new_parts_delta.each do |new_part_delta|
+        if new_part_delta.text
+          last_existing_part = existing_parts.last
+          if last_existing_part&.text
+            last_existing_part.text << new_part_delta.text
+            @io << new_part_delta.text if @io.respond_to?(:<<)
+          else
+            existing_parts << new_part_delta
+            @io << new_part_delta.text if @io.respond_to?(:<<)
+          end
+        elsif new_part_delta.functionCall
+          existing_parts << new_part_delta
         end
       end
     end

data/lib/llm/providers/gemini.rb

@@ -20,6 +20,7 @@ module LLM
   class Gemini < Provider
     require_relative "gemini/response/embedding"
     require_relative "gemini/response/completion"
+    require_relative "gemini/response/web_search"
     require_relative "gemini/error_handler"
     require_relative "gemini/format"
     require_relative "gemini/stream_parser"
@@ -125,6 +126,31 @@ module LLM
       "gemini-2.5-flash"
     end
 
+    ##
+    # @note
+    #  This method includes certain tools that require configuration
+    #  through a set of options that are easier to set through the
+    #  {LLM::Provider#tool LLM::Provider#tool} method.
+    # @see https://ai.google.dev/gemini-api/docs/google-search Gemini docs
+    # @return (see LLM::Provider#tools)
+    def tools
+      {
+        google_search: tool(:google_search),
+        code_execution: tool(:code_execution),
+        url_context: tool(:url_context)
+      }
+    end
+
+    ##
+    # A convenience method for performing a web search using the
+    # Google Search tool.
+    # @param query [String] The search query.
+    # @return [LLM::Response] The response from the LLM provider.
+    def web_search(query:)
+      complete(query, tools: [tools[:google_search]])
+        .extend(LLM::Gemini::Response::WebSearch)
+    end
+
     private
 
     def headers

data/lib/llm/providers/openai/format.rb

@@ -45,7 +45,11 @@ class LLM::OpenAI
     # @return [Hash]
     def format_tools(params)
       tools = params.delete(:tools)
-      (tools.nil? || tools.empty?) ? {} : {tools: tools.map { _1.format(self) }}
+      if tools.nil? || tools.empty?
+        {}
+      else
+        {tools: tools.map { _1.respond_to?(:format) ? _1.format(self) : _1 }}
+      end
     end
   end
 end

data/lib/llm/providers/openai/response/responds.rb

@@ -2,21 +2,35 @@
 
 module LLM::OpenAI::Response
   module Responds
-    def outputs = [format_message]
-    def choices = body.output
-    def tools = output.select { _1.type == "function_call" }
+    def model = body.model
+    def response_id = respond_to?(:response) ? response["id"] : id
+    def choices = [format_message]
+    def annotations = choices[0].annotations
+
+    def prompt_tokens = body.usage&.input_tokens
+    def completion_tokens = body.usage&.output_tokens
+    def total_tokens = body.usage&.total_tokens
+
+    ##
+    # Returns the aggregated text content from the response outputs.
+    # @return [String]
+    def output_text
+      choices.find(&:assistant?).content || ""
+    end
 
     private
 
     def format_message
       message = LLM::Message.new("assistant", +"", {response: self, tool_calls: []})
-      choices.each.with_index do |choice, index|
+      output.each.with_index do |choice, index|
         if choice.type == "function_call"
           message.extra[:tool_calls] << format_tool(choice)
         elsif choice.content
           choice.content.each do |c|
             next unless c["type"] == "output_text"
             message.content << c["text"] << "\n"
+            next unless c["annotations"]
+            message.extra["annotations"] = [*message.extra["annotations"], *c["annotations"]]
           end
         end
       end

data/lib/llm/providers/openai/response/web_search.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module LLM::OpenAI::Response
+  ##
+  # The {LLM::OpenAI::Response::WebSearch LLM::OpenAI::Response::WebSearch}
+  # module provides methods for accessing web search results from a web search
+  # tool call made via the {LLM::Provider#web_search LLM::Provider#web_search}
+  # method.
+  module WebSearch
+    ##
+    # Returns one or more search results
+    # @return [Array<LLM::Object>]
+    def search_results
+      LLM::Object.from_hash(
+        choices[0]
+          .annotations
+          .map { _1.slice(:title, :url) }
+      )
+    end
+  end
+end

data/lib/llm/providers/openai/responses/stream_parser.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+class LLM::OpenAI
+  ##
+  # @private
+  class Responses::StreamParser
+    ##
+    # Returns the fully constructed response body
+    # @return [LLM::Object]
+    attr_reader :body
+
+    ##
+    # @param [#<<] io An IO-like object
+    # @return [LLM::OpenAI::Responses::StreamParser]
+    def initialize(io)
+      @body = LLM::Object.new(output: []) # Initialize with an empty output array
+      @io = io
+    end
+
+    ##
+    # @param [Hash] chunk
+    # @return [LLM::OpenAI::Responses::StreamParser]
+    def parse!(chunk)
+      tap { handle_event(chunk) }
+    end
+
+    private
+
+    def handle_event(chunk)
+      case chunk["type"]
+      when "response.created"
+        chunk.each do |k, v|
+          next if k == "type"
+          @body[k] = v
+        end
+        @body.output ||= []
+      when "response.output_item.added"
+        output_index = chunk["output_index"]
+        item = LLM::Object.from_hash(chunk["item"])
+        @body.output[output_index] = item
+        @body.output[output_index].content ||= []
+      when "response.content_part.added"
+        output_index = chunk["output_index"]
+        content_index = chunk["content_index"]
+        part = LLM::Object.from_hash(chunk["part"])
+        @body.output[output_index] ||= LLM::Object.new(content: [])
+        @body.output[output_index].content ||= []
+        @body.output[output_index].content[content_index] = part
+      when "response.output_text.delta"
+        output_index = chunk["output_index"]
+        content_index = chunk["content_index"]
+        delta_text = chunk["delta"]
+        output_item = @body.output[output_index]
+        if output_item&.content
+          content_part = output_item.content[content_index]
+          if content_part && content_part.type == "output_text"
+            content_part.text ||= ""
+            content_part.text << delta_text
+            @io << delta_text if @io.respond_to?(:<<)
+          end
+        end
+      when "response.output_item.done"
+        output_index = chunk["output_index"]
+        item = LLM::Object.from_hash(chunk["item"])
+        @body.output[output_index] = item
+      when "response.content_part.done"
+        output_index = chunk["output_index"]
+        content_index = chunk["content_index"]
+        part = LLM::Object.from_hash(chunk["part"])
+        @body.output[output_index] ||= LLM::Object.new(content: [])
+        @body.output[output_index].content ||= []
+        @body.output[output_index].content[content_index] = part
+      end
+    end
+  end
+end

data/lib/llm/providers/openai/responses.rb

@@ -15,6 +15,7 @@ class LLM::OpenAI
   #   [res1, res2].each { llm.responses.delete(_1) }
   class Responses
     require_relative "response/responds"
+    require_relative "responses/stream_parser"
     include Format
 
     ##
@@ -37,12 +38,13 @@ class LLM::OpenAI
     def create(prompt, params = {})
       params = {role: :user, model: @provider.default_model}.merge!(params)
       params = [params, format_schema(params), format_tools(params)].inject({}, &:merge!).compact
-      role = params.delete(:role)
+      role, stream = params.delete(:role), params.delete(:stream)
+      params[:stream] = true if stream.respond_to?(:<<) || stream == true
       req = Net::HTTP::Post.new("/v1/responses", headers)
       messages = [*(params.delete(:input) || []), LLM::Message.new(role, prompt)]
       body = JSON.dump({input: [format(messages, :response)].flatten}.merge!(params))
       set_body_stream(req, StringIO.new(body))
-      res = execute(request: req)
+      res = execute(request: req, stream:, stream_parser:)
       LLM::Response.new(res).extend(LLM::OpenAI::Response::Responds)
     end
 
@@ -86,5 +88,9 @@ class LLM::OpenAI
       name = "JSONSchema"
       {text: {format: {type: "json_schema", name:, schema:}}}
     end
+
+    def stream_parser
+      LLM::OpenAI::Responses::StreamParser
+    end
   end
 end

data/lib/llm/providers/openai/stream_parser.rb

@@ -39,36 +39,44 @@ class LLM::OpenAI
     def merge_choices!(choices)
       choices.each do |choice|
         if @body.choices[choice["index"]]
-          target = @body["choices"][choice["index"]]["message"]
+          target_message = @body["choices"][choice["index"]]["message"]
           delta = choice["delta"]
           delta.each do |key, value|
-            if target[key]
-              if key == "content"
-                target[key] << value
-                @io << value if @io.respond_to?(:<<)
-              elsif key == "tool_calls"
-                merge_tools!(target, value)
-              else
-                target[key] = value
-              end
-            else
+            if key == "content"
+              target_message[key] ||= +""
+              target_message[key] << value
               @io << value if @io.respond_to?(:<<)
-              target[key] = value
+            elsif key == "tool_calls"
+              merge_tools!(target_message, value)
+            else
+              target_message[key] = value
             end
           end
         else
-          target = {"message" => {"role" => "assistant"}}
-          @body["choices"][choice["index"]] = target
-          target["message"].merge!(choice["delta"])
+          message_hash = {"role" => "assistant"}
+          @body["choices"][choice["index"]] = {"message" => message_hash}
+          choice["delta"].each do |key, value|
+            if key == "content"
+              @io << value if @io.respond_to?(:<<)
+              message_hash[key] = value
+            else
+              message_hash[key] = value
+            end
+          end
         end
       end
     end
 
     def merge_tools!(target, tools)
+      target["tool_calls"] ||= []
       tools.each.with_index do |toola, index|
         toolb = target["tool_calls"][index]
-        if toolb
-          toola["function"].each { toolb["function"][_1] << _2 }
+        if toolb && toola["function"] && toolb["function"]
+          # Append to existing function arguments
+          toola["function"].each do |func_key, func_value|
+            toolb["function"][func_key] ||= +""
+            toolb["function"][func_key] << func_value
+          end
         else
           target["tool_calls"][index] = toola
         end

data/lib/llm/providers/openai/vector_stores.rb

@@ -3,9 +3,19 @@
 class LLM::OpenAI
   ##
   # The {LLM::OpenAI::VectorStores LLM::OpenAI::VectorStores} class provides
-  # an interface for [OpenAI's vector stores API](https://platform.openai.com/docs/api-reference/vector_stores/create)
+  # an interface for [OpenAI's vector stores API](https://platform.openai.com/docs/api-reference/vector_stores/create).
+  #
+  # @example
+  #  llm = LLM.openai(key: ENV["OPENAI_SECRET"])
+  #  files = %w(foo.pdf bar.pdf).map { llm.files.create(file: _1) }
+  #  store = llm.vector_stores.create(name: "PDF Store", file_ids: files.map(&:id))
+  #  store = llm.vector_stores.poll(vector: store)
+  #  print "[-] store is ready", "\n"
+  #  chunks = llm.vector_stores.search(vector: store, query: "What is Ruby?")
+  #  chunks.each { |chunk| puts chunk }
   class VectorStores
     require_relative "response/enumerable"
+    PollError = Class.new(LLM::Error)
 
     ##
     # @param [LLM::Provider] provider
@@ -181,6 +191,27 @@ class LLM::OpenAI
       LLM::Response.new(res)
     end
 
+    ##
+    # Poll a vector store until its status is "completed"
+    # @param [String, #id] vector The ID of the vector store
+    # @param [Integer] attempts The current number of attempts (default: 0)
+    # @param [Integer] max The maximum number of iterations (default: 50)
+    # @raise [LLM::PollError] When the maximum number of iterations is reached
+    # @return [LLM::Response]
+    def poll(vector:, attempts: 0, max: 50)
+      if attempts == max
+        raise LLM::PollError, "vector store '#{vector.id}' has status '#{vector.status}' after #{max} attempts"
+      elsif vector.status == "expired"
+        raise LLM::PollError, "vector store '#{vector.id}' has expired"
+      elsif vector.status != "completed"
+        vector = get(vector:)
+        sleep(0.1 * (2**attempts))
+        poll(vector:, attempts: attempts + 1, max:)
+      else
+        vector
+      end
+    end
+
     private
 
     [:headers, :execute, :set_body_stream].each do |m|

data/lib/llm/providers/openai.rb

@@ -16,6 +16,7 @@ module LLM
   class OpenAI < Provider
     require_relative "openai/response/embedding"
     require_relative "openai/response/completion"
+    require_relative "openai/response/web_search"
     require_relative "openai/error_handler"
     require_relative "openai/format"
     require_relative "openai/stream_parser"
@@ -146,6 +147,37 @@ module LLM
       "gpt-4.1"
     end
 
+    ##
+    # @note
+    #  This method includes certain tools that require configuration
+    #  through a set of options that are easier to set through the
+    #  {LLM::Provider#tool LLM::Provider#tool} method.
+    # @return (see LLM::Provider#tools)
+    def tools
+      {
+        web_search: tool(:web_search),
+        file_search: tool(:file_search),
+        image_generation: tool(:image_generation),
+        code_interpreter: tool(:code_interpreter),
+        computer_use: tool(:computer_use)
+      }
+    end
+
+    ##
+    # A convenience method for performing a web search using the
+    # OpenAI web search tool.
+    # @example
+    #   llm = LLM.openai(key: ENV["KEY"])
+    #   res = llm.web_search(query: "summarize today's news")
+    #   res.search_results.each { |item| print item.title, ": ", item.url, "\n" }
+    # @param query [String] The search query.
+    # @return [LLM::Response] The response from the LLM provider.
+    def web_search(query:)
+      responses
+        .create(query, store: false, tools: [tools[:web_search]])
+        .extend(LLM::OpenAI::Response::WebSearch)
+    end
+
     private
 
     def headers

data/lib/llm/tool.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+##
+# The {LLM::Tool LLM::Tool} class represents a platform-native tool
+# that can be activated by an LLM provider. Unlike {LLM::Function LLM::Function},
+# these tools are pre-defined by the provider and their capabilities
+# are already known to the underlying LLM.
+#
+# @example
+#   #!/usr/bin/env ruby
+#   llm = LLM.gemini ENV["KEY"]
+#   bot = LLM::Bot.new(llm, tools: [LLM.tool(:google_search)])
+#   bot.chat("Summarize today's news", role: :user)
+#   print bot.messages.find(&:assistant?).content, "\n"
+class LLM::Tool < Struct.new(:name, :options, :provider)
+  ##
+  # @return [String]
+  def to_json(...)
+    to_h.to_json(...)
+  end
+
+  ##
+  # @return [Hash]
+  def to_h
+    case provider.class.to_s
+    when "LLM::Anthropic" then options.merge("name" => name.to_s)
+    when "LLM::Gemini" then {name => options}
+    else options.merge("type" => name.to_s)
+    end
+  end
+  alias_method :to_hash, :to_h
+end

data/lib/llm/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module LLM
-  VERSION = "0.14.2"
+  VERSION = "0.16.0"
 end

data/lib/llm.rb

@@ -18,6 +18,7 @@ module LLM
   require_relative "llm/function"
   require_relative "llm/eventstream"
   require_relative "llm/eventhandler"
+  require_relative "llm/tool"
 
   module_function
 
@@ -38,7 +39,7 @@ module LLM
   end
 
   ##
-  # @param (see LLM::Provider#initialize)
+  # @param key (see LLM::Provider#initialize)
   # @return (see LLM::Ollama#initialize)
   def ollama(key: nil, **)
     require_relative "llm/providers/ollama" unless defined?(LLM::Ollama)
@@ -79,7 +80,7 @@ module LLM
   end
 
   ##
-  # Define a function
+  # Define or get a function
   # @example
   #   LLM.function(:system) do |fn|
   #     fn.description "Run system command"
@@ -94,7 +95,11 @@ module LLM
   # @param [Proc] b The block to define the function
   # @return [LLM::Function] The function object
   def function(name, &b)
-    functions[name.to_s] = LLM::Function.new(name, &b)
+    if block_given?
+      functions[name.to_s] = LLM::Function.new(name, &b)
+    else
+      functions[name.to_s]
+    end
   end
 
   ##

data/llm.gemspec

@@ -40,4 +40,5 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "standard", "~> 1.50"
   spec.add_development_dependency "vcr", "~> 6.0"
   spec.add_development_dependency "dotenv", "~> 2.8"
+  spec.add_development_dependency "net-http-persistent", "~> 4.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: llm.rb
 version: !ruby/object:Gem::Version
-  version: 0.14.2
+  version: 0.16.0
 platform: ruby
 authors:
 - Antar Azri
@@ -150,6 +150,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.8'
+- !ruby/object:Gem::Dependency
+  name: net-http-persistent
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
 description: llm.rb is a zero-dependency Ruby toolkit for Large Language Models that
   includes OpenAI, Gemini, Anthropic, xAI (grok), DeepSeek, Ollama, and LlamaCpp.
   The toolkit includes full support for chat, streaming, tool calling, audio, images,
@@ -170,6 +184,7 @@ files:
 - lib/llm/bot/prompt/completion.rb
 - lib/llm/bot/prompt/respond.rb
 - lib/llm/buffer.rb
+- lib/llm/client.rb
 - lib/llm/error.rb
 - lib/llm/eventhandler.rb
 - lib/llm/eventstream.rb
@@ -193,6 +208,7 @@ files:
 - lib/llm/providers/anthropic/response/completion.rb
 - lib/llm/providers/anthropic/response/enumerable.rb
 - lib/llm/providers/anthropic/response/file.rb
+- lib/llm/providers/anthropic/response/web_search.rb
 - lib/llm/providers/anthropic/stream_parser.rb
 - lib/llm/providers/deepseek.rb
 - lib/llm/providers/deepseek/format.rb
@@ -211,6 +227,7 @@ files:
 - lib/llm/providers/gemini/response/files.rb
 - lib/llm/providers/gemini/response/image.rb
 - lib/llm/providers/gemini/response/models.rb
+- lib/llm/providers/gemini/response/web_search.rb
 - lib/llm/providers/gemini/stream_parser.rb
 - lib/llm/providers/llamacpp.rb
 - lib/llm/providers/ollama.rb
@@ -240,7 +257,9 @@ files:
 - lib/llm/providers/openai/response/image.rb
 - lib/llm/providers/openai/response/moderations.rb
 - lib/llm/providers/openai/response/responds.rb
+- lib/llm/providers/openai/response/web_search.rb
 - lib/llm/providers/openai/responses.rb
+- lib/llm/providers/openai/responses/stream_parser.rb
 - lib/llm/providers/openai/stream_parser.rb
 - lib/llm/providers/openai/vector_stores.rb
 - lib/llm/providers/xai.rb
@@ -256,6 +275,7 @@ files:
 - lib/llm/schema/object.rb
 - lib/llm/schema/string.rb
 - lib/llm/schema/version.rb
+- lib/llm/tool.rb
 - lib/llm/utils.rb
 - lib/llm/version.rb
 - llm.gemspec