llm.rb 15.0.3 → 15.2.0

data/README.md

@@ -1,7 +1,7 @@
 <p align="center">
   <a href="https://r.uby.dev">
     <img
-      src="https://github.com/r-uby-dev/llm.rb/raw/main/rubydev.svg"
+      src="rubydev.svg"
       width="400"
       height="200"
       border="0"
@@ -10,18 +10,16 @@
   </a>
 </p>
 
-> A [r.uby.dev](https://r.uby.dev/llm) project.
+> [r.uby.dev](https://r.uby.dev/llm) project.
 
 Welcome to the canonical llm.rb repository.
 
 llm.rb is an advanced runtime for building agentic AI applications
 on CRuby. It has zero runtime dependencies by default, supports
 concurrent and parallel tool execution and has a single coherent API
-that spans 13+ providers. Streaming, tools, guards, compaction, the
-REPL, builtin MCP/A2A support and the database integrations all build
-on the same three concepts: providers, contexts, and agents.
+that spans 14+ providers.
 
-The most effective way to learn about llm.rb is to ask [the r.uby.dev chatbot](https://r.uby.dev)
+The easiest way to learn about llm.rb is to ask [the r.uby.dev chatbot](https://r.uby.dev)
 a question. It is connected to the llm.rb GitHub repository, backed by
 ActiveRecord and uses the builtin MCP feature to connect to GitHub. All
 answers are grounded in the llm.rb source code.
@@ -108,8 +106,8 @@ class MyStream < LLM::Stream
   def on_skill_return(agent, skill, result)
   end
 
-  # A request was rate limited and will be retried.
-  def on_rate_limit(error)
+  # A request was rate limited or timed out and will be retried.
+  def on_retry(error, attempt)
   end
 end
 
@@ -204,6 +202,12 @@ with the `:fork` and `:ractor` strategies. The
 The `:fork` strategy also provides a separate process that offers
 isolation from its parent.
 
+A couple of concurrency strategies require optional, opt-in dependencies.
+The `async` strategy requires the [async](https://github.com/socketry/async)
+gem and the `fork` strategy requires the [xchan.rb](https://github.com/r-uby-dev/xchan.rb)
+gem. The `fiber` strategy requires a scheduler (`Fiber.scheduler`) but by
+default Ruby does not provide one.
+
 ```ruby
 require "llm"
 require "llm/tools"
@@ -241,37 +245,35 @@ end
 ```
 </details>
 <details>
-<summary>Console (<code>binding.pry</code> for agents)</summary>
+<summary>Console (<code>binding.irb</code> for agents)</summary>
 <br>
 
-The [LLM::Agent#repl](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#repl-instance_method)
-method drops you into a highly capable read-eval-print loop (REPL)
+The [LLM::Agent#console](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#console-instance_method)
+method drops you into a highly capable interactive console
 that is built on top of curses. It can help you debug agents,
 test your tools, connect to MCP servers, and even A2A agents.
-The REPL stands out because it connects to the surrounding
+The console stands out because it connects to the surrounding
 runtime and it can be extended by your code. Think of it as
-`binding.pry` but for agents.
+`binding.irb` but for agents.
 
 ##### Demo
 
-[Watch in high quality on asciinema](https://asciinema.org/a/OsS8wwaasKasoDDz)
-
-![llm.rb REPL demo](demo.gif)
+![llm.rb console demo](demo.gif)
 
 
 ##### Installation
 
-The REPL is distributed with llm.rb so you don't have to install
+The console is distributed with llm.rb so you don't have to install
 a separate gem but it requires a number of optional dependencies
 to be installed separately. The following gems provide the full
 experience:
 
-    gem install curses kramdown xchan.rb test-cmd.rb
+    gem install unicode-display_width curses kramdown xchan.rb test-cmd.rb
 
 ##### Persistence
 
-The `path:` option can be set on an agent for automatic persistence
-across REPL sessions. The `tools:` option attaches extra tools
+the `path:` option can be set on an agent for automatic persistence
+across console sessions. The `tools:` option attaches extra tools
 for the duration of the session. Recall previous turns with Ctrl+P and
 Ctrl+N.
 
@@ -281,21 +283,26 @@ require "llm/tools"
 
 llm = LLM.deepseek(key: ENV["KEY"])
 agent = LLM::Agent.new(llm, name: "my-agent", path: "agent.json")
-agent.repl(tools: LLM::Tool.subclasses)
+agent.console(tools: LLM::Tool.subclasses)
 ```
 
 ##### CLI
 
 The `llm.rb` executable is available on your PATH after installation.
-It starts a REPL session from any directory.The CLI auto-detects your
+It starts a console session from any directory. The CLI auto-detects your
 provider from standard environment variables (`DEEPSEEK_API_KEY`,
 `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.). Persistent sessions are
 stored under `~/.llm.rb/` and restored automatically on your next visit.
 
 ```bash
-llm.rb                     # auto-detect from $DEEPSEEK_API_KEY
+llm.rb                     # auto-detect from $PROVIDER_API_KEY
 llm.rb -p openai           # use OpenAI explicitly
+llm.rb -m gpt-5.6          # use a model other than the provider default
+llm.rb -c thread           # run tool calls on a separate thread
+llm.rb -n curb             # use libcurl as the HTTP transport
+llm.rb -x 900              # read timeout of 15 minutes
 llm.rb -t                  # temporary session, no persistence
+llm.rb -v                  # print the version
 ```
 </details>
 <details>
@@ -335,53 +342,90 @@ for both Rack-based / Rails-based applications. On databases
 where it is supported, such as PostgreSQL, the column can be optimized by using
 the `jsonb` type.
 
+The following example is based on the agent used to power the
+[r.uby.dev chatbot](https://r.uby.dev).
+
 ```ruby
 require "active_record"
 require "llm"
 require "llm/active_record"
 
-class Email < ApplicationRecord
-  acts_as_agent do |agent|
-    agent.set name: "mail",
-              instructions: "Write concise, friendly replies to emails",
-              model: "deepseek-v4-pro"
+class Raven < ActiveRecord::Base
+  acts_as_agent(format: :jsonb) do |agent|
+    agent.set name: "raven",
+              description: "a chatbot for the r.uby.dev website",
+              instructions: proc { File.read(File.join(__dir__, "raven", "prompt.md")) },
+              tools: :tools,
+              concurrency: :async
   end
 
-  def draft_reply!
-    talk("Draft a reply to:\n\n#{body}")
+  def research_issues
+    talk("research open pull requests on r-uby-dev/llm")
   end
 
-  def summarize
-    talk("Summarize this email thread in a few sentences")
+  def research_codebase
+    talk("research the codebase on r-uby-dev/llm")
   end
 
-  private
+  ##
+  # @return [LLM::MCP]
+  def github
+    @github ||= LLM::MCP.http(
+      url: "https://api.githubcopilot.com/mcp/",
+      headers: {"Authorization" => "Bearer #{ENV['GITHUB_RUBYDEV_PAT']}"},
+      transport: :net_http_persistent
+    )
+  end
 
   ##
-  # By convention, this method defines the provider for a model.
-  # If necessary, it can be renamed with: provider: :your_method.
+  # @return [Array<LLM::Tool>]
+  def tools
+    github.tools.select { allowlist.include?(_1.name.to_s) }
+  end
+
+  private
+
   def set_provider
-    LLM.deepseek(key: ENV["KEY"])
+    LLM.deepseek
   end
 
-  ##
-  # By convention, this method returns the context options given
-  # to LLM::Context or LLM::Agent. This method can be left undefined.
-  def set_context
-    {}
+  def allowlist
+    %w[
+        get_commit
+        get_file_contents
+        list_branches
+        list_commits
+        search_code
+        search_commits
+        search_repositories
+        search_issues
+        pull_request_read
+        list_pull_requests
+        list_issues
+        issue_read
+    ].freeze
   end
 end
 
-email = Email.create!(subject: "Streaming support", body: "How do I stream responses?")
-email.draft_reply!
+agent = Raven.create!
 
 ##
-# The conversation (the email and the draft
-# reply) is persisted to the email's column. A
-# fresh instance restores it and continues the
-# thread, so the summary below knows what was
-# already drafted:
-Email.find(email.id).summarize
+# Every call to `talk` automatically persists
+# to the database (under the hood research_issues
+# calls the talk method)
+agent.research_issues
+
+##
+# The conversation was persisted to database. A
+# fresh instance restores it and continues where
+# we left off
+agent = Raven.find(agent.id).tap(&:research_codebase)
+
+##
+# Start an agent console.
+# Query agent's state, debug, etc.
+# The console does not persist back to the database.
+agent.console
 ```
 </details>
 
@@ -491,15 +535,15 @@ the call, or `nil` to let it run:
 ```ruby
 class PolicyGuard < LLM::Guard
   def call(function:)
-    if function.name == "shell"
+    if function.name == "exec"
       function.return(error: true, type: "policy_error",
-                      message: "shell is disabled")
+                      message: "exec is disabled")
     end
   end
 end
 
 llm = LLM.deepseek(key: ENV["KEY"])
-agent = LLM::Agent.new(llm, tools: [Shell, ReadFile], guard: PolicyGuard)
+agent = LLM::Agent.new(llm, tools: [LLM::Tool::Exec, ReadFile], guard: PolicyGuard)
 ```
 </details>
 
@@ -566,7 +610,8 @@ agent.talk "Hello"
 
 Rate-limited requests are retried automatically by default. Agents
 retry a 429 up to five times with a growing backoff before giving
-up, so most request failures resolve on their own. Set `retry_budget`
+up, so most request failures resolve on their own. Connection and
+read timeouts are retried the same way. Set `retry_budget`
 to change the number of retries, or `retry_budget: 0` to disable
 them.
 
@@ -585,21 +630,28 @@ agent.talk "Hello"
 <summary>Observability</summary>
 <br>
 
-Trace what an agent is doing by attaching a tracer. Hook into
-requests, tool calls, and other runtime events to debug a
-misbehaving agent, monitor latency, or export spans to an
-observability backend. All built-in tracers share one interface,
-so switching between them means changing a class name:
+It is possible to trace what an agent is doing by attaching a
+tracer. A tracer can hook into requests, tool calls, and other
+runtime events to debug an agent, provide insights, monitor latency,
+or export spans to an observability backend. All built-in tracers
+share one interface, so switching between them means changing a
+factory method:
 
-* [`LLM::Tracer::PrettyLogger`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer/PrettyLogger.html): human-readable single-line logs to stderr, ideal during development.
-* [`LLM::Tracer::Telemetry`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer/Telemetry.html):
+* [`LLM::Tracer.pretty_logger`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html#pretty_logger-class_method): human-readable single-line logs to stderr, ideal during development.
+* [`LLM::Tracer.telemetry`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html#telemetry-class_method):
 exports spans via OTLP for OpenTelemetry in production.
-* [`LLM::Tracer::Logger`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer/Logger.html):
+* [`LLM::Tracer.logger`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html#logger-class_method):
 structured JSON to stdout or a file.
 
+It is also possible to create your own tracer by creating a subclass
+of [`LLM::Tracer`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html)
+that implements a number of callbacks that cover an agent's lifecycle.
+The tracer feature provides visibility into what the runtime is doing,
+and the tracer API lets other code hook into that feature.
+
 ```ruby
 llm = LLM.deepseek(key: ENV["KEY"])
-agent = LLM::Agent.new(llm, tracer: LLM::Tracer::PrettyLogger.new(llm))
+agent = LLM::Agent.new(llm, tracer: LLM::Tracer.pretty_logger(llm))
 agent.talk "Hello"
 ```
 </details>
@@ -624,7 +676,7 @@ class Agent < LLM::Agent
   set name: "sysadmin",
       description: "system administration agent",
       model: "deepseek-v4-pro",
-      tools: [LLM::Tool::Shell]
+      tools: [LLM::Tool::Exec]
 end
 
 llm = LLM.deepseek(key: ENV["KEY"])
@@ -653,6 +705,7 @@ change.
 * **xAI** (`LLM.xai`)
 * **Z.ai** (`LLM.zai`)
 * **Moonshot (Kimi)** (`LLM.moonshot`)
+* **OpenRouter** (`LLM.openrouter`)
 * **Alibaba (Qwen3)** (`LLM.alibaba`, also `LLM.aliyun`)
 * **Mistral** (`LLM.mistral`)
 * **AWS Bedrock** (`LLM.bedrock`)
@@ -674,6 +727,7 @@ llm = LLM.anthropic
 llm = LLM.deepseek
 llm = LLM.alibaba  # also: LLM.aliyun
 llm = LLM.moonshot
+llm = LLM.openrouter
 llm = LLM.mistral
 ```
 </details>
@@ -691,6 +745,7 @@ llm = LLM.anthropic(key: ENV["ANTHROPIC_API_KEY"])
 llm = LLM.deepseek(key: ENV["DEEPSEEK_API_KEY"])
 llm = LLM.alibaba(key: ENV["DASHSCOPE_API_KEY"]) # also: LLM.aliyun
 llm = LLM.moonshot(key: ENV["MOONSHOT_API_KEY"])
+llm = LLM.openrouter(key: ENV["OPENROUTER_API_KEY"])
 llm = LLM.mistral(key: ENV["MISTRAL_API_KEY"])
 ```
 </details>
@@ -709,7 +764,7 @@ require "llm"
 
 llm      = LLM.openai
 registry = llm.registry                # => LLM::Provider#registry
-cheapest = registry.models.sort.first  # => LLM::Model
+cheapest = registry.models.sort.first  # => LLM::Registry::Model
 cheapest.id                            # => "text-embedding-3-small"
 cheapest.context_window                # => 8191
 cheapest.structured_output?            # => false
@@ -734,6 +789,51 @@ llm = LLM.deepseek(
 ```
 </details>
 
+<details>
+<summary>Timeouts</summary>
+<br>
+
+Providers accept two timeouts:
+
+* `connect_timeout` - opening the connection. Defaults to 5 seconds.
+* `read_timeout` - waiting for a response on an idle connection.
+  Defaults to 600 seconds (10 minutes).
+
+The longer read timeout leaves room for slow reasoning models and
+local models. The legacy `timeout:` option remains as a shorthand for
+`read_timeout`. Timeouts are retriable:
+[`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html)
+retries a timed out request up to its
+[`retry_budget`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#retry_budget-class_method)
+(five by default), so a dropped connection or a slow first token
+is often something we can recover from.
+
+```ruby
+llm = LLM.deepseek(
+  connect_timeout: 5,   # opening the connection
+  read_timeout: 600     # waiting for the next bytes
+)
+```
+</details>
+
+<details>
+<summary>Headers</summary>
+<br>
+
+Providers can accept a custom set of headers with
+the [`LLM::Provider#with`](https://r.uby.dev/api-docs/llm.rb/LLM/Provider.html#with-instance_method) method.
+For example, you could set a custom User-Agent header,
+or provide headers that carry special meaning to
+certain providers (eg OpenAI, OpenRouter).
+
+```ruby
+llm = LLM.openrouter
+llm = llm.with("HTTP-Referer" => "https://example.com")
+llm = llm.with("X-OpenRouter-Title" => "Example App")
+```
+
+</details>
+
 ### RAG
 
 Most providers offer an embedding model that can be
@@ -803,6 +903,17 @@ IO.copy_stream res.images[0], "rocket-with-dog.svg"
 
 ## FAQ
 
+<details>
+<summary>Where can I see llm.rb in action?</summary>
+<br>
+<p>
+
+The [r.uby.dev](https://r.uby.dev) website is powered
+by llm.rb and its builtin MCP feature. It is connected
+to this very GitHub repository. It is designed to help
+you learn and troubleshoot llm.rb.
+</p>
+</details>
 <details>
 <summary>What about local LLM support?</summary>
 <br>
@@ -870,7 +981,7 @@ than three years and over that time multiple other
 contributors have contributed to llm.rb as well. New
 contributors are always welcome.
 
-I use the repl that is distributed with llm.rb to build
+I use the console that is distributed with llm.rb to build
 llm.rb itself so there is a healthy feedback loop and
 llm.rb has also been battle tested in production
 environments.
@@ -883,13 +994,19 @@ I am constantly focused on improving llm.rb by using
 it as my primary driver for development.
 </details>
 
-## Resources
+## See also
+
+The [roda-llm](https://github.com/r-uby-dev/roda-llm#readme) project
+is how I deploy multiple ActiveRecord-backed llm.rb agents over HTTP.
+Each agent has an identical interface at a unique path that provide
+CRUD operations and stream support (via SSE - Server Side Events).
+It lets you focus on implementing agents rather than the glue that
+brings them together. It is implemented as a Roda plugin that could
+be hosted within a Rails application or other Rack-based applications.
 
-The [r.uby.dev chatbot](https://r.uby.dev) is connected
-to this very GitHub repository. It can read documentation,
-source code, issues, and pull requests. The [docs/](docs/)
-directory contains the full documentation and the chatbot
-can find the answers to your questions there.
+The [docs/](docs/) directory contains the full documentation and
+the chatbot can find the answers to your questions there. Or you
+can read them yourself.
 
 ## License
 

data/bin/llm.rb

@@ -42,6 +42,10 @@ def wrap(text, width)
   end
 end
 
+def version
+  warn "llm.rb v#{LLM::VERSION}"
+end
+
 def help
   prog = File.basename($PROGRAM_NAME)
   warn ""
@@ -49,16 +53,21 @@ def help
   warn ""
   warn "Options:"
   warn "  -p PROVIDER    Choose a provider"
+  warn "  -m MODEL       Choose a model"
   warn "  -c STRATEGY    Concurrency strategy for tool calls (eg thread, async, fork)"
   warn "  -n TRANSPORT   HTTP transports - net-http (default), net-http-persistent, and curb"
+  warn "  -x TIMEOUT     The default read timeout (in seconds)"
   warn "  -t             Temporary session that doesn't persist to disk"
+  warn "  -v             Print version information"
   warn "  -h             Show this help"
   warn ""
   warn "Examples:"
   warn "  #{prog}                     # auto-detect provider from $PROVIDER_API_KEY"
   warn "  #{prog} -p openai           # use OpenAI"
+  warn "  #{prog} -m gpt-5.6          # use a model other than the provider default"
   warn "  #{prog} -n curb             # use libcurl"
   warn "  #{prog} -c thread           # run tool calls on a separate thread"
+  warn "  #{prog} -x 900              # read timeout of 15mins"
   warn "  #{prog} -h                  # this help"
   warn ""
 end
@@ -70,12 +79,12 @@ def loaderror(ex)
   warn ""
   warn_title "Missing dependency: #{gem}"
   warn ""
-  warn "     The repl needs this gem, but it's not installed."
+  warn "     The console needs this gem, but it's not installed."
   warn ""
   wrapped "Fix:  gem install #{gem}", "     "
   wrapped "Or:   bundle add #{gem}", "     "
   warn ""
-  warn "     Tip:  If you don't need the repl, you can use the"
+  warn "     Tip:  If you don't need the console, you can use the"
   wrapped "library directly with:  require \"llm\"", "           "
   warn ""
   warn "  ───────────────────────────────────────────────────────"
@@ -94,6 +103,11 @@ def fatal(ex)
     wrapped detail, "     "
   end
   warn ""
+  warn "     Backtrace:"
+  ex.backtrace.drop(1).first(3).each do |line|
+    wrapped line, "       "
+  end
+  warn ""
   warn "     This is an unexpected error. If it keeps happening,"
   warn "     consider opening an issue at"
   warn "     https://github.com/r-uby-dev/llm/issues"
@@ -111,7 +125,7 @@ def main(argv)
   # Make sure the dependencies are satisified first
   begin
     require "llm/tools"
-    require "llm/repl"
+    require "llm/console"
   rescue LLM::LoadError => ex
     loaderror(ex)
     exit 1
@@ -126,6 +140,9 @@ def main(argv)
       when '-h'
         help
         exit 0
+      when '-v'
+        version
+        exit 0
       when '-t'
         temp = true
       when '-c'
@@ -153,6 +170,22 @@ def main(argv)
           help
           exit 1
         end
+      when '-m'
+        model = argv.shift
+        if model.nil?
+          warn "llm.rb: -m switch requires an argument"
+          help
+          exit 1
+        end
+      when '-x'
+        timeout = argv.shift
+        if timeout.nil?
+          warn "llm.rb: -x switch requires an argument"
+          help
+          exit 1
+        else
+          timeout = Integer(timeout)
+        end
       else
         warn "llm.rb: unknown option #{option}"
         help
@@ -164,9 +197,10 @@ def main(argv)
     # No provider has been given.
     # Try to infer one.
     transport ||= :net_http
+    options = timeout ? {timeout:, transport:} : {transport:}
     if provider.nil?
       llm = providers.filter_map do
-        LLM.method(_1).call(transport:)
+        LLM.method(_1).call(**options)
       rescue ArgumentError
       end.first
       if llm.nil?
@@ -175,7 +209,7 @@ def main(argv)
       end
     else
       begin
-        llm = LLM.method(provider).call(transport:)
+        llm = LLM.method(provider).call(**options)
       rescue ArgumentError
         warn "llm.rb: set credentials for #{provider}"
         exit 1
@@ -209,12 +243,21 @@ def main(argv)
       File.binwrite file, JSON.pretty_generate(data)
     end
 
+    if model
+      if not llm.registry.keys.include?(model)
+        warn "llm.rb: #{model} is not a valid #{llm.name} model"
+        exit 1
+      end
+    else
+      model = llm.default_model
+    end
+
     ##
     # Let's go!
     concurrency ||= :sequential
     path  = temp ? nil : data[Dir.getwd]
-    agent = LLM::Agent.new(llm, path:, concurrency:, tools: LLM::Tool.subclasses)
-    agent.repl
+    agent = LLM::Agent.new(llm, model:, path:, concurrency:, tools: LLM::Tool.subclasses)
+    agent.console
   rescue Interrupt
     warn "llm.rb: Bye!"
   rescue => ex