llm.rb 15.3.0 → 15.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a461691c878ffef8343a8f1ae2dad4298ebb631d379d901110ef8ab8641d9c68
-  data.tar.gz: 95bb85e3debb178ebbe6949cd3cafbb3625d6f20a72d617f08f1a600354911ba
+  metadata.gz: de1f4eeee4762548e75eac2b5d4112de49cf446944b30c545b76e62750aad6c3
+  data.tar.gz: 4abc1d2d707717fde266b48bfa626f06d553514761653db8c3cd190cfcffb864
 SHA512:
-  metadata.gz: c9b6e723d1bc16d9cc515a9ae2a3fd34072ce88024804d5b7b0e713cff0b08c26e78e2008641763bbc9a3aabea6f262023fa1f243d9fe155f8d8fdf58215d24f
-  data.tar.gz: e125d9936d34b3550fb504d281760b73bc617aa4b5d42f1f4d1e3b9b3edd98a448c4a7509d010c62b67b1f57def5f3a9f1e088786b860e6d8f08c94bf64aa0f8
+  metadata.gz: c28cadd2a412e2623b472d3be7814aa51a462076782fc847bc73cdc57a601a630d909bb9b40c29d31dd7107b487bf169bdeb245e9803581df0d90d642694bc16
+  data.tar.gz: 2fd3c1905d2db4212958e98e537db76d37f0f30f5cee4dda3cbf351c7f0fffe1ce6eaa5b38553e80b9441b029e2edf78c4e9ea75c200b731638dbbe49bd947df

data/CHANGELOG.md

@@ -17,6 +17,123 @@
 
 *No unreleased changes yet. Check back after the next release.*
 
+## v15.4.0
+
+Changes since `v15.3.0`.
+
+This release saves token usage with the context state, moves the retry budget
+onto the provider, groups a turn's spans under one trace, and lets an agent
+declare `name`, `description`, `path`, and `tool_budget` with a block. It also
+gives every tracer request an id, adds `LLM::ActiveRecord::Message`, reads
+`AGENTS.md` as the console's system prompt, removes `LLM::Guard::Loop`, and
+refreshes the model registry.
+
+### Core
+
+* **context: save token usage with the state** <br>
+  [`LLM::Context#to_h`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#to_h-instance_method)
+  now writes `context_used` and `context_window`, so a saved state can be
+  inspected without loading the runtime. They are never read back, and a payload
+  without them still loads.
+
+### Provider
+
+* **provider: decide the retry budget on the provider** <br>
+  [`LLM::Provider#retry_budget`](https://r.uby.dev/api-docs/llm.rb/LLM/Provider.html#retry_budget-instance_method)
+  returns how many times a rate-limited request is retried before the error is
+  raised, and defaults to 5. An agent that sets no `retry_budget` of its own
+  now takes the budget from its provider instead of special-casing Alibaba, so
+  [`LLM::Alibaba`](https://r.uby.dev/api-docs/llm.rb/LLM/Alibaba.html)
+  still retries 8 times and a provider that recovers from rate limits slowly can
+  return a higher budget. An explicit `retry_budget:` still takes precedence.
+
+### Agent
+
+* **agent: group a turn's spans into one trace** <br>
+  [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) now opens a
+  `llm.turn` trace group around every turn, so all spans a turn produces share
+  one trace id. It uses the agent's tracer when it has one, the provider's
+  otherwise; previously a provider-wide tracer split one turn across traces.
+
+* **agent: declare `name`, `description`, `path`, and `tool_budget` with a block** <br>
+  A block passed to any of these four class-level setters was previously
+  ignored, because the call was read as a getter. Now
+  [`LLM::Agent.name`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#name-class_method)
+  and [`LLM::Agent.description`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#description-class_method)
+  store the block, and an instance resolves it against itself, or against its
+  ORM record when it is bound to one. `name` and `description` also accept a
+  `Symbol` or a `Proc`, and the class-level reader still returns what was
+  configured, so read them on an instance for the resolved string.
+
+### Console
+
+* **console: use `AGENTS.md` as the system prompt** <br>
+  `bin/llm.rb` now looks for `AGENTS.md` in the current working directory when
+  it boots, and when the file exists its contents become the agent's
+  instructions for the session. The instructions are injected once, so a
+  resumed session that already has a system message keeps the one it has.
+
+* **console: stop a cancelled turn from crashing the console** <br>
+  A turn cancelled with Esc can still have chunks queued, and they arrive after
+  the buffer that renders them is closed. The streaming path then tried to
+  replace a row that no longer existed and raised.
+  [`LLM::Console::Buffer`](https://r.uby.dev/api-docs/llm.rb/LLM/Console/Buffer.html)
+  now reports whether it has a row to replace through `open?`, the stream drops
+  chunks that arrive for a closed buffer, and `replace` keeps the current text
+  instead of raising.
+
+### Guard
+
+* **guard: remove `LLM::Guard::Loop`** <br>
+  `LLM::Guard::Loop` is removed. It stopped repeated tool-call patterns, but
+  could also interrupt a loop that was making progress, so it did not hold up as
+  a default. Agents no longer enable a guard of their own; bound the loop with
+  `tool_budget` instead.
+
+### ActiveRecord
+
+* **activerecord: add `LLM::ActiveRecord::Message`** <br>
+  [`LLM::ActiveRecord::Message`](https://r.uby.dev/api-docs/llm.rb/LLM/ActiveRecord/Message.html)
+  is a virtual model over the JSONB column that holds an agent's state, so
+  messages can be filtered, ordered, and counted in SQL instead of in memory.
+  `for(agent:)` returns an
+  [`ActiveRecord::Relation`](https://api.rubyonrails.org/classes/ActiveRecord/Relation.html)
+  with `id`, `role`, `content`, `tools`, and `position` columns; a row's
+  `unwrap!` returns the message. The class never materializes as a table.
+
+### Tracer
+
+* **tracer: give every request an id** <br>
+  [`LLM::Tracer#on_request_start`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html#on_request_start-instance_method)
+  now takes a `request_id:`, a UUIDv7 that the runtime mints when a request
+  begins and passes to `on_request_finish` and `on_request_error` for that same
+  request. A turn can make many requests, so `trace_group_id` groups a turn but
+  not the events inside one request; the id lets a tracer correlate a request's
+  start, finish, and error events. The keyword is required, so a subclass that
+  overrides these hooks must accept it.
+
+* **tracer: record the model the provider answers with** <br>
+  [`LLM::Tracer::Telemetry`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer/Telemetry.html)
+  now sets `gen_ai.response.model` from the model on the response instead of the
+  model requested, so a span reports the model a provider actually served. A
+  router model such as `openrouter/auto` records the model it resolved to,
+  while `gen_ai.request.model` keeps the name that was asked for.
+
+### Registry
+
+* **refresh model metadata** <br>
+  Update `data/` with current pricing, limits, and capabilities for the
+  Alibaba, Anthropic, Bedrock, DeepInfra, DeepSeek, Mistral, OpenAI,
+  OpenRouter, xAI, and Z.ai registries. Claude Opus 5.5 reaches Anthropic,
+  Bedrock, and OpenRouter, OpenAI adds GPT-6 Sol and GPT-6 Luna, Bedrock adds
+  Gemma 4 and more regional Claude Sonnet 4 and GPT-5.6 entries, and
+  OpenRouter adds the Xiaomi MiMo V2.6, Nex N2.5, Cohere Command A+, and
+  Qwen3.8 Omni Flash models. Z.ai adds `glm-4.6v-flash` and `glm-5.3-flashx`,
+  xAI adds Grok 4.7, DeepSeek adds a `low` reasoning effort to
+  `deepseek-v4-pro` and deprecates `deepseek-v4-flash` and
+  `deepseek-v4-flash-vision-exp`, DeepInfra reprices `tencent/Hy3`, and
+  OpenRouter drops `anthropic/claude-opus-4` and `kwaipilot/kat-coder-pro-v2`.
+
 ## v15.3.0
 
 Changes since `v15.2.2`.
@@ -290,9 +407,10 @@ and a `-v` switch to the CLI, and refreshes the model registry.
   The shared [`LLM::Tool::Utils`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool/Utils.html)
   module now requires the `test-cmd.rb` gem (at `~> 2.7.1`) itself and
   exposes the `spawn` and `wait` helpers, so any tool that includes
-  `Utils` gets command spawning without requiring `exec` directly. The
-  `Git`, `Mkdir`, `Rg`, `Ruby`, `Exec`, and `Bundle` tools all
-  inherit their bounded-output protections from this shared runner.
+  `Utils` gets command spawning without requiring `exec` directly.
+  `Exec` and `ReadFile` include `Utils`, and the tools that shell out
+  (`Git`, `Rg`, `Mkdir`, `Ruby`, and `Bundle`) route through `Exec`,
+  so they all get the same bounded output.
 
 * **tools: route `git`, `rg`, `mkdir`, and `ruby` through `exec`** <br>
   `LLM::Tool::Git`, `LLM::Tool::Rg`, `LLM::Tool::Mkdir`, and

data/README.md

@@ -52,7 +52,7 @@ an invalid state that would lead to API-level errors. For example,
 when a tool call is interrupted it could leave an unanswered tool
 call that a model will reject on the next turn. The runtime takes
 care of this by pruning orphaned tool calls and ensuring that the
-tool loop always remains valid. 
+tool loop always remains valid.
 
 ```ruby
 require "llm"
@@ -141,7 +141,9 @@ call them on your behalf, and they're one of the most powerful features
 for extending the feature set or abilities of a model.
 
 The runtime also ships with a catalog of built-in tools for
-filesystem, search, and shell operations.
+filesystem, search, and shell operations, and providers expose
+platform-native tools such as web search and code execution that run
+on the provider's side.
 
 ```ruby
 class ReadFile < LLM::Tool
@@ -276,13 +278,18 @@ for agents.
 
 ##### Installation
 
-The console is distributed with llm.rb so you don't have to install
-a separate gem but it requires a number of optional dependencies
-to be installed separately. The following gems provide the full
-experience:
+The console is distributed with llm.rb but it requires a number
+of optional dependencies to be installed separately. The following
+gems provide the full experience:
 
     gem install unicode-display_width curses kramdown xchan.rb test-cmd.rb
 
+For convenience it is also possible to just use the following, it
+is a metagem that depends on llm.rb and all the dependencies it requires
+to run the console:
+
+    gem install llm-shell
+
 ##### Persistence
 
 the `path:` option can be set on an agent for automatic persistence
@@ -363,21 +370,27 @@ require "active_record"
 require "llm"
 require "llm/active_record"
 
-class Raven < ActiveRecord::Base
+##
+# The Robert agent.
+class Robert < ActiveRecord::Base
   acts_as_agent(format: :jsonb) do |agent|
-    agent.set name: "raven",
-              description: "a chatbot for the r.uby.dev website",
-              instructions: proc { File.read(File.join(__dir__, "raven", "prompt.md")) },
+    agent.set name: "robert",
+              description: "robert is an agent that has access to the official " \
+                            "r.uby.dev GitHub repositories. He can access the repositories " \
+                            "to answer your question(s) about r.uby.dev projects.",
+              instructions: proc { File.read(File.join(__dir__, "robert", "prompt.md")) },
               tools: :tools,
-              concurrency: :async
-  end
+              concurrency: :async,
 
-  def research_issues
-    talk("research open pull requests on r-uby-dev/llm")
-  end
+              ##
+              # The maximum number of tool calls per-turn.
+              tool_budget: 25,
 
-  def research_codebase
-    talk("research the codebase on r-uby-dev/llm")
+              ##
+              # The default tracer that all agents have associated
+              # with them. The tracer exports a trace to a couple of
+              # SQL tables.
+              tracer: proc { Raven::Tracer::SQL.new(llm, agent: self) }
   end
 
   ##
@@ -398,10 +411,6 @@ class Raven < ActiveRecord::Base
 
   private
 
-  def set_provider
-    LLM.deepseek
-  end
-
   def allowlist
     %w[
         get_commit
@@ -420,19 +429,18 @@ class Raven < ActiveRecord::Base
   end
 end
 
-agent = Raven.create!
+agent = Robert.create!
 
 ##
 # Every call to `talk` automatically persists
-# to the database (under the hood research_issues
-# calls the talk method)
-agent.research_issues
+# to the database.
+agent.talk "what's new on the llm.rb repository?"
 
 ##
 # The conversation was persisted to database. A
 # fresh instance restores it and continues where
 # we left off
-agent = Raven.find(agent.id).tap(&:research_codebase)
+agent = Robert.find(agent.id).talk "and what about roda-llm?"
 
 ##
 # Start an agent console.
@@ -440,6 +448,100 @@ agent = Raven.find(agent.id).tap(&:research_codebase)
 # The console does not persist back to the database.
 agent.console
 ```
+</details>
+<details>
+<summary> SQL optimizations </summary>
+<br>
+
+In a database environment the runtime optimizes for
+the PostgreSQL database and its builtin support for
+the `jsonb` column type. An agent fits in a single
+column, on a single row, and that column carries
+everything it has done: messages, tool calls,
+context usage, and so on. It works well in practice
+and means you can store an agent almost anywhere.
+
+For scenarios where performance matters most the runtime
+ships with virtual ActiveRecord classes that never materialize
+in your database but provide a SQL view into the column where
+an agent stores its runtime state. They return
+[`ActiveRecord::Relation`](https://api.rubyonrails.org/classes/ActiveRecord/Relation.html)
+objects, so the filtering happens in the database.
+
+```ruby
+class Agent < ActiveRecord::Base
+  acts_as_agent(format: :jsonb) do |agent|
+    agent.set name: "activerecord agent"
+  end
+end
+
+##
+# Find an instance of your agent
+agent = Agent.find_by(id: 1)
+
+##
+# Returns a relation over the agent's messages.
+# It is scoped to the agent, and it yields one
+# instance of LLM::ActiveRecord::Message per
+# message the agent has produced.
+messages = LLM::ActiveRecord::Message.for(agent:)
+
+##
+# The relation chains like any other
+messages.where(role: "assistant")
+        .order(position: :desc)
+        .limit(10)
+
+##
+# Count, too
+messages.count
+```
+
+**Schema**
+
+Each row carries a message, flattened into columns:
+
+| column | contents |
+| --- | --- |
+| `agent_id` | the agent a message belongs to |
+| `id` | the message id |
+| `role` | the message role |
+| `content` | the message content |
+| `tools` | the tool calls a message carries |
+| `position` | the position of a message in the conversation |
+| `data` | the whole message, as the runtime stores it |
+
+**Indexes**
+
+The queries the view runs are already covered. They expand
+one agent, found by primary key, so they are index scans.
+There is nothing to add for
+`LLM::ActiveRecord::Message.for(agent:)`.
+
+The queries you write on top of it are not. Once a question
+is asked of every agent, the column is expanded row by row
+and no index helps the view itself. Index the column for
+those questions instead:
+
+```sql
+CREATE INDEX index_agents_on_data
+  ON agents USING gin (data jsonb_path_ops);
+
+CREATE INDEX index_agents_on_context_used
+  ON agents (((data ->> 'context_used')::int));
+```
+
+The first serves containment (`@>`) and path queries over
+the state as a whole. The second serves a scalar key, and
+the runtime already writes `context_used` and
+`context_window` at the top level, so "sessions over 80%
+full" becomes cheap. Both assume `format: :jsonb`.
+
+**However:** an agent's whole conversation lives in one
+value, so every save rewrites it, and a GIN index is
+maintained with it. Prefer an index on a key or two over
+the whole column.
+
 </details>
 
 <details><summary>MCP</summary>
@@ -534,10 +636,9 @@ even answer for it. Because it runs before the tool, anything
 it intercepts never executes. Policy, validation, quotas, and
 cost ceilings all live here.
 
-[`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html)
-enables
-[`LLM::Guard::Loop`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard/Loop.html)
-by default, so agents get loop protection out of the box. To
+Agents and contexts use
+[`LLM::Guard::Null`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard/Null.html)
+by default, so a guard only runs when you configure one. To
 write your own guard, subclass
 [`LLM::Guard`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard.html)
 and implement
@@ -737,11 +838,16 @@ is also distributed with llm.rb.
 ```ruby
 llm = LLM.openai
 llm = LLM.anthropic
+llm = LLM.google
 llm = LLM.deepseek
-llm = LLM.alibaba  # also: LLM.aliyun
+llm = LLM.deepinfra
+llm = LLM.xai
+llm = LLM.zai
 llm = LLM.moonshot
 llm = LLM.openrouter
+llm = LLM.alibaba  # also: LLM.aliyun
 llm = LLM.mistral
+llm = LLM.bedrock
 ```
 </details>
 <details>
@@ -755,10 +861,14 @@ key at all.
 ```ruby
 llm = LLM.openai(key: ENV["OPENAI_API_KEY"])
 llm = LLM.anthropic(key: ENV["ANTHROPIC_API_KEY"])
+llm = LLM.google(key: ENV["GOOGLE_API_KEY"])
 llm = LLM.deepseek(key: ENV["DEEPSEEK_API_KEY"])
-llm = LLM.alibaba(key: ENV["DASHSCOPE_API_KEY"]) # also: LLM.aliyun
+llm = LLM.deepinfra(key: ENV["DEEPINFRA_API_KEY"])
+llm = LLM.xai(key: ENV["XAI_API_KEY"])
+llm = LLM.zai(key: ENV["ZHIPU_API_KEY"])
 llm = LLM.moonshot(key: ENV["MOONSHOT_API_KEY"])
 llm = LLM.openrouter(key: ENV["OPENROUTER_API_KEY"])
+llm = LLM.alibaba(key: ENV["DASHSCOPE_API_KEY"]) # also: LLM.aliyun
 llm = LLM.mistral(key: ENV["MISTRAL_API_KEY"])
 ```
 </details>
@@ -919,22 +1029,18 @@ IO.copy_stream res.images[0], "rocket-with-dog.svg"
 <details>
 <summary>Where can I see llm.rb in action?</summary>
 <br>
-<p>
 
-The [r.uby.dev](https://r.uby.dev) website is powered
-by llm.rb and its builtin MCP feature. It is connected
-to this very GitHub repository. It is designed to help
-you learn and troubleshoot llm.rb.
-</p>
+The [r.uby.dev](https://r.uby.dev) website deploys
+multiple llm.rb agents that guests can interact with
+and there is even an agent that is connected to this
+GitHub repository.
 </details>
 <details>
 <summary>What about local LLM support?</summary>
 <br>
-<p>
+
 The following providers can be run used with models that
-are running on your own hardware. They're reasonably well
-tested but not my main driver:
-</p>
+are running on your own hardware.
 
 * Ollama
 * Llamacpp
@@ -943,29 +1049,27 @@ tested but not my main driver:
 <details>
 <summary>I have a limited budget. What should I do?</summary>
 <br>
-<p>
+
 There are a few options. The first option is to host
 your own model, and use the ollama or llamacpp
 providers. This can be difficult though because
 a capable model requires hardware that can
 match it. If you have the ability to self-host,
 this would be my first option.
-</p>
-<p>
+
 The second option is DeepSeek. <br>
 The deepseek-v4-flash model costs pennies to use. <br>
 And llm.rb has been optimized for deepseek. For example,
 DeepSeek does not have image generation capabilities
 but on the llm.rb runtime it does (vector graphics only,
 though).
-</p>
-<p>
+
 The same is true for structured outputs. DeepSeek does
 not support structured outputs in the same way as OpenAI or
 Google, but the llm.rb runtime makes it appear as
 though it does, through the `json_object` response
 type.
-</p>
+
 If you're on a budget, DeepSeek is hard to beat.
 </details>
 <details>
@@ -988,23 +1092,35 @@ web</a>.
 <summary>Who maintains llm.rb?</summary>
 <br>
 
-The llm.rb project is maintained primarily by one
-person. llm.rb has been in active development for more
-than three years and over that time multiple other
-contributors have contributed to llm.rb as well. New
-contributors are always welcome.
-
-I use the console that is distributed with llm.rb to build
-llm.rb itself so there is a healthy feedback loop and
-llm.rb has also been battle tested in production
-environments.
+The llm.rb project was started more than three
+years ago by
+[@0x1eef](https://github.com/0x1eef) and
+[@antaz](https://github.com/0x1eef). The primary
+maintainer is [@0x1eef](https://github.com/0x1eef).
+Over those three years multiple other contributors have
+contributed to llm.rb as well, and new contributors are
+always welcome.
+</details>
 
-I have also also written llm.rb agents within the
-repository that help me maintain the documentation,
-and backport changes to the mruby-llm runtime as well.
+<details>
+<summary>How well tested is llm.rb?</summary>
+<br>
 
-I am constantly focused on improving llm.rb by using
-it as my primary driver for development.
+It is battle tested daily.
+
+The console that is distributed with llm.rb is used
+to build llm.rb so there is a healthy, active feedback
+loop. It also powers the [r.uby.dev](https://r.uby.dev)
+website where multiple llm.rb agents are deployed with
+the help of [roda-llm](https://github.com/r-uby-dev/roda-llm).
+I'm also aware of at least one production Rails deployment
+at a large-ish company.
+
+And this git repository includes llm.rb agents that help me
+maintain the documentation and perform other repository
+maintainence. The feedback loop is constant. Outside of that
+there is a large test suite that covers live requests (recorded
+by VCR) and database interactions.
 </details>
 
 ## See also
@@ -1019,7 +1135,7 @@ be hosted within a Rails application or other Rack-based applications.
 
 The [docs/](docs/) directory contains the full documentation and
 the chatbot can find the answers to your questions there. Or you
-can read them yourself.
+can read them yourself. :)
 
 ## License
 

data/bin/llm.rb

@@ -193,6 +193,12 @@ def main(argv)
       end
     end
 
+    ##
+    # Let's use `AGENTS.md` as the system prompt
+    agent_opts = {}
+    sysprompt  = File.join(Dir.getwd, "AGENTS.md")
+    File.file?(sysprompt) ? agent_opts.merge!(instructions: File.read(sysprompt)) : {}
+
     ##
     # No provider has been given.
     # Try to infer one.
@@ -255,8 +261,9 @@ def main(argv)
     ##
     # Let's go!
     concurrency ||= :sequential
-    path  = temp ? nil : data[Dir.getwd]
-    agent = LLM::Agent.new(llm, model:, path:, concurrency:, tools: LLM::Tool.subclasses)
+    path   = temp ? nil : data[Dir.getwd]
+    params = agent_opts.merge(model:, path:, concurrency:, tools: LLM::Tool.subclasses)
+    agent  = LLM::Agent.new(llm, params)
     agent.console
   rescue Interrupt
     warn "llm.rb: Bye!"

data/data/alibaba.json

@@ -1193,6 +1193,51 @@
         "output_audio": 15.11
       }
     },
+    "kimi-k3": {
+      "id": "kimi-k3",
+      "name": "Kimi K3",
+      "description": "Multimodal Kimi model with 1M context and toggleable max-effort thinking for long-horizon agent work",
+      "family": "kimi-k3",
+      "attachment": true,
+      "reasoning": true,
+      "reasoning_options": [
+        {
+          "type": "effort",
+          "values": [
+            "low",
+            "high",
+            "max"
+          ]
+        }
+      ],
+      "tool_call": true,
+      "interleaved": {
+        "field": "reasoning_content"
+      },
+      "structured_output": true,
+      "temperature": false,
+      "release_date": "2026-07-16",
+      "last_updated": "2026-07-16",
+      "modalities": {
+        "input": [
+          "text",
+          "image"
+        ],
+        "output": [
+          "text"
+        ]
+      },
+      "open_weights": true,
+      "limit": {
+        "context": 1048576,
+        "output": 1048576
+      },
+      "cost": {
+        "input": 3,
+        "output": 15,
+        "cache_read": 0.3
+      }
+    },
     "qwen2-5-vl-72b-instruct": {
       "id": "qwen2-5-vl-72b-instruct",
       "name": "Qwen2.5-VL 72B Instruct",