llm.rb 13.1.0 → 14.0.0

data/README.md

@@ -15,30 +15,153 @@
 Welcome to the canonical llm.rb repository.
 
 llm.rb is an advanced runtime for building capable AI applications
-on CRuby. By default it has zero runtime dependencies although certain
-functionality (such as ActiveRecord support) require
+on CRuby. It has zero runtime dependencies by default, and a single
+coherent API that spans 12+ providers. Streaming, tools, guards,
+compaction, the REPL, builtin MCP/A2A support and the database
+integrations all build on the same three concepts: providers,
+contexts, and agents.
+
+Once you learn the fundamentals, everything else falls into place
+naturally. Some features, such as ActiveRecord support, require
 optional dependencies that are opt-in.
 
-When you want to learn more than what the README covers, checkout
-the [deepdive.md](https://r.uby.dev/llm/deepdive/).
-
 ## Features
 
-The runtime supports OpenAI, OpenAI-compatible endpoints, Anthropic, Google
-Gemini, Mistral, DeepSeek, DeepInfra, xAI, Z.ai, AWS Bedrock, Ollama, and llama.cpp.
-It has first-class support for streaming, tool calls,  MCP
-and A2A, embeddings, vector stores, OCR, context compaction,
-and the RAG pattern.
+One runtime, 12+ providers. The same API drives OpenAI, Anthropic,
+Google Gemini, Moonshot (kimi), Mistral, DeepSeek, DeepInfra, xAI, Z.ai,
+AWS Bedrock, Ollama, and llama.cpp, so switching models or providers
+can be done with minimal code change.
+
+<details>
+<summary><b>Agents</b></summary>
+
+* **First-class support** <br>
+  llm.rb is designed to build agents. They can be attached to a
+  terminal-based read-eval-print loop (repl), persisted to disk
+  or a database column, run tools concurrently and be safely
+  interrupted.
+
+* **Builtin REPL** <br>
+  A curses-based TUI for talking to an agent interactively. It
+  renders markdown, shows a live status line with context usage
+  and running cost, and recalls previous turns, so a
+  conversation survives a restart.
+
+* **Persistence** <br>
+  Set `path:` and the agent saves its conversation to disk
+  automatically. ActiveRecord and Sequel support keep the same
+  state in a single database column, so you pick the storage
+  and the API stays identical.
+
+</details>
+
+<details>
+<summary><b>MCP &amp; A2A</b></summary>
+
+* **MCP** <br>
+  The Model Context Protocol is first-class. Point an MCP client
+  at any tool server over stdio or HTTP, and its tools translate
+  into local `LLM::Tool` subclasses, with the same tracing and
+  error handling.
+
+* **A2A** <br>
+  The Agent 2 Agent protocol is first-class. Point an A2A client
+  at another agent over HTTP or JSON-RPC, and call its skills
+  exactly like local tools.
 
-There are multiple HTTP backends to choose from, tools can be run concurrently
-or in parallel via threads, async tasks, fibers, ractors, and fork, and it is
-also possible to make a tool call while the model is still streaming.
+</details>
+
+<details>
+<summary><b>ORM</b></summary>
+
+* **ActiveRecord** <br>
+  Add `acts_as_agent` to a model and the agent state lives in a
+  single database column, saved after every turn and restored
+  on load. Works in Rack and Rails apps, with `jsonb` on
+  PostgreSQL.
+
+* **Sequel** <br>
+  Add `plugin :agent` to a Sequel model for the same single-
+  column persistence, with the `pg_json` extension loaded
+  automatically on PostgreSQL.
+
+</details>
+
+<details>
+<summary><b>RAG</b></summary>
+
+* **RAG, out of the box** <br>
+  Embeddings, OCR, and OpenAI's vector stores API come first-
+  class. Ground answers in your own documents, with vectors in
+  a managed store or in your own database such as sqlite-vec
+  or pgvector.
+
+</details>
+
+<details>
+<summary><b>Runtime</b></summary>
 
-The runtime builds on top of three core concepts: providers, contexts, and agents,
-so once you learn the fundamentals, everything else falls into place naturally. And once
-you learn llm.rb, you will also be able to use
-<a href="https://r.uby.dev/mruby-llm">mruby-llm</a> and
-<a href="https://r.uby.dev/wasm-llm">wasm-llm</a> because the API is pretty much identical.
+* **Streaming** <br>
+  Streaming is first-class, with structured callbacks for
+  content, reasoning, and tool calls. Tools can start while
+  the model is still talking, so the first result lands
+  before the response finishes.
+
+* **Concurrency** <br>
+  Six ways to run tools: sequential, threads, async, fibers,
+  forks, and ractors. Plus three HTTP backends, so you pick
+  the concurrency model that fits the workload, not the other
+  way around.
+
+* **Interruption** <br>
+  Cancel an in-flight request or a running tool at any moment,
+  on any transport or concurrency strategy. A stuck call never
+  leaves a thread running that you can't stop.
+
+</details>
+
+<details>
+<summary><b>Provider extras</b></summary>
+
+* **DeepSeek-optimized** <br>
+  DeepSeek is the most cost-effective option for API users, and the
+  runtime closes its gaps: [`LLM::Schema`](https://r.uby.dev/api-docs/llm.rb/LLM/Schema.html)
+  makes structured outputs work despite no official API, and
+  `images.create`/`edit` produce SVG vector graphics.
+</details>
+
+<details>
+<summary><b>Portable</b></summary>
+
+* **mruby-llm** <br>
+  The same runtime runs on mruby as
+  [mruby-llm](https://github.com/r-uby-dev/mruby-llm), with an
+  almost identical interface and the same set of capabilities.
+
+</details>
+
+<details>
+<summary><b>Everything else</b></summary>
+
+* **Skills** <br>
+  Write a SKILL.md, get a tool. The runtime spawns a
+  disposable subagent with the skill's instructions and tool
+  set for one turn, then discards it. Fresh and stateless
+  every call.
+
+* **A unified plugin family** <br>
+  Compactors, transformers, and guards all share one
+  interface. Context management, message rewriting, and tool
+  supervision (policy, quotas, loop detection) plug in the
+  same way and compose freely.
+
+* **Cost and usage tracking** <br>
+  Every context tracks its own cost and token usage, per turn.
+  Break the spend down by input, output, cache, and reasoning,
+  so the exact cost of any conversation is visible at a
+  glance.
+
+</details>
 
 ## Install
 
@@ -54,8 +177,9 @@ The
 [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html)
 class is the default high-level interface,
 and it is recommended for most use-cases. It manages tool execution
-automatically, guards against infinite loops, manages conversation
-state, and much more.
+automatically and
+[guards against infinite loops](https://r.uby.dev/llm/deepdive/advanced/guard),
+manages conversation state, and much more.
 
 ```ruby
 require "llm"
@@ -71,7 +195,8 @@ agent.talk "Hello world"
 is a class-level DSL that accepts a Hash of properties. Each key resolves to a
 corresponding class accessor: `name`, `description`, `model`, `tools`,
 `instructions`, `schema`, `stream`, `tracer`, `concurrency`, `confirm`,
-`path`, and `skills`. All options are optional; zero or more can be set.
+`path`, `skills`, and `tool_budget`. All options are optional; zero or
+more can be set.
 An error is raised for unknown keys so that typos are caught early.
 
 ```ruby
@@ -123,6 +248,11 @@ sometimes that can be useful, but usually for advanced use-cases.
 If you're new to llm.rb, try
 [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) first.
 
+Every context tracks its own token usage and estimated cost. After any
+turn, you can read the cost breakdown through
+[`LLM::Context#cost`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#cost-instance_method),
+and the REPL shows the running total in the status line.
+
 ```ruby
 require "llm"
 
@@ -140,6 +270,11 @@ an optional set of typed parameters. <br> The model can choose to
 call them on your behalf, and they're one of the most powerful features
 for extending the feature set or abilities of a model.
 
+The runtime also ships with a catalog of built-in tools for
+filesystem, search, and shell operations. See the
+[deepdive.md](https://r.uby.dev/llm/deepdive/fundamentals/builtin_tools)
+for details.
+
 ```ruby
 class ReadFile < LLM::Tool
   name "read-file"
@@ -153,12 +288,39 @@ class ReadFile < LLM::Tool
 end
 ```
 
+##### set
+
+[`LLM::Tool.set`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html#set-class_method)
+is an alternative way to define tool properties using a Hash. It works
+the same way as
+[`LLM::Agent.set`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#set-class_method)
+and accepts the same keys that the individual methods do: `name`,
+`description`, `parameters`, `required`, and `defaults`:
+
+```ruby
+class MathTool < LLM::Tool
+  set name: "math",
+      description: "Performs arithmetic",
+      parameters: [
+        [:x, Integer, "first number" , {required: true}],
+        [:y, Integer, "second number", {default: 0}]
+      ]
+
+  def call(x:, y: 0)
+    {result: x + y}
+  end
+end
+```
+
 #### LLM::Stream
 
 Streams can be simple IO objects or subclasses of
 [`LLM::Stream`](https://r.uby.dev/api-docs/llm.rb/LLM/Stream.html)
 with structured callbacks for content,
 reasoning, tool calls, tool returns, and compaction.
+Streams can also observe message transformers, which rewrite
+outgoing messages before they reach the provider (see the
+[deepdive.md](https://r.uby.dev/llm/deepdive/advanced/transformer)).
 
 ```ruby
 class MyStream < LLM::Stream
@@ -180,9 +342,12 @@ agent.talk "Explain Ruby fibers."
 
 [`LLM::Schema`](https://r.uby.dev/api-docs/llm.rb/LLM/Schema.html)
 subclasses produce typed, structured
-output from any model call. Pass a schema to `LLM::Context#talk`,
-`LLM::Agent#talk`, or `LLM::Provider#complete` to receive validated
-JSON instead of free text. Schemas work alongside tools and streams.
+output from any model call. Pass a schema to
+[`LLM::Context#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#talk-instance_method),
+[`LLM::Agent#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#talk-instance_method),
+or
+[`LLM::Provider#complete`](https://r.uby.dev/api-docs/llm.rb/LLM/Provider.html#complete-instance_method)
+to receive validated JSON instead of free text. Schemas work alongside tools and streams.
 
 [`LLM::Schema`](https://r.uby.dev/api-docs/llm.rb/LLM/Schema.html)
 can define objects, arrays, enums, nested schemas,
@@ -217,10 +382,16 @@ res.content!  # => {city: "Paris", temperature: 15.0, conditions: "Cloudy"}
 
 The [LLM::Agent#repl](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#repl-instance_method)
 method drops you into a curses-based TUI for talking to an
-agent interactively. Set `path:` on the agent for automatic
-persistence across REPL sessions. The `tools:` option attaches
-extra tools for the duration of the session. It is like
-`binding.pry` but for agents. For the full reference see the
+agent interactively. It renders markdown directly in the
+terminal and shows a live status line with context usage,
+running cost, and the current tool call. A second thread keeps
+the UI responsive while the model works. Think of it as
+`binding.pry` but for agents.
+
+Set `path:` on the agent for automatic persistence across REPL
+sessions. The `tools:` option attaches extra tools for the
+duration of the session. Recall previous turns with Ctrl+P and
+Ctrl+N. For the full reference, see the
 [REPL section](https://r.uby.dev/llm/deepdive/fundamentals/repl) in the
 deepdive.
 
@@ -268,6 +439,22 @@ agent = LLM::Agent.new(llm, stream: $stdout, tools: mcp.tools)
 agent.talk "Run the tool"
 ```
 
+##### Persistent connections
+
+Set `persistent: true` on HTTP transports to reuse connections
+across requests. This uses
+[`Net::HTTP::Persistent`](https://github.com/drbrain/net-http-persistent)
+under the hood and avoids opening a new TCP connection for every
+request:
+
+```ruby
+mcp = LLM::MCP.http(
+  url: "https://api.githubcopilot.com/mcp/",
+  headers: {"Authorization" => "Bearer #{ENV.fetch('GITHUB_PAT')}"},
+  persistent: true
+)
+```
+
 #### LLM::A2A
 
 The Agent 2 Agent (A2A) protocol has first-class support
@@ -287,6 +474,52 @@ agent = LLM::Agent.new(llm, stream: $stdout, tools: a2a.skills)
 agent.talk "Run the skill"
 ```
 
+##### Persistent connections
+
+Set `persistent: true` on HTTP transports to reuse connections
+across requests. This uses
+[`Net::HTTP::Persistent`](https://github.com/drbrain/net-http-persistent)
+under the hood and avoids opening a new TCP connection for every
+request:
+
+```ruby
+a2a = LLM::A2A.rest(url: "https://agent.example.com", persistent: true)
+a2a = LLM::A2A.jsonrpc(url: "https://agent.example.com", persistent: true)
+```
+
+#### LLM::Guard
+
+[`LLM::Guard`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard.html)
+is the hook that sees every tool call before it runs. A guard
+can let a call through, cancel it, block it with an error, or
+even answer for it. Because it runs before the tool, anything
+it intercepts never executes. Policy, validation, quotas, and
+cost ceilings all live here.
+
+[`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html)
+enables
+[`LLM::Guard::Loop`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard/Loop.html)
+by default, so agents get loop protection out of the box. To
+write your own guard, subclass
+[`LLM::Guard`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard.html)
+and implement
+[`LLM::Guard#call`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard.html#call-instance_method).
+The pending call arrives as `function:`. Return a value to close
+the call, or `nil` to let it run:
+
+```ruby
+class PolicyGuard < LLM::Guard
+  def call(function:)
+    if function.name == "shell"
+      function.return(error: true, type: "policy_error",
+                      message: "shell is disabled")
+    end
+  end
+end
+
+agent = LLM::Agent.new(llm, guard: PolicyGuard)
+```
+
 #### LLM::Skill
 
 A skill turns a markdown file into a callable tool. When the model
@@ -296,7 +529,7 @@ one turn and returns the result, then is discarded. Each call
 is fresh and stateless. For a deeper explanation see the
 [deepdive.md](https://r.uby.dev/llm/deepdive/fundamentals/skills).
 
-**SKILL.md**
+##### SKILL.md
 
 ```markdown
 ---
@@ -309,7 +542,7 @@ Collect the recent git log, analyze each commit,
 and write a summary to summary.txt.
 ```
 
-**agent.rb**
+##### agent.rb
 
 ```ruby
 require "llm"
@@ -330,7 +563,8 @@ or PostgreSQL's [pg-vector](https://github.com/pgvector/pgvector).
 
 llm.rb also includes support for OpenAI's vector store API. It
 provides a vector database as a HTTP service but we won't cover
-that here.
+that here. For a deeper explanation see the
+[deepdive.md](https://r.uby.dev/llm/deepdive/fundamentals/embeddings).
 
 ```ruby
 require "llm"
@@ -417,6 +651,44 @@ agent = Agent.create!
 agent.talk "perform research"
 ```
 
+#### Images
+
+A handful of providers can generate images from a text prompt.
+OpenAI, Google, xAI, and DeepInfra all support it. The API is
+the same across providers:
+
+```ruby
+require "llm"
+
+llm = LLM.openai(key: ENV["KEY"])
+res = llm.images.create(prompt: "a dog on a rocket to the moon")
+IO.copy_stream res.images[0], "rocket.png"
+```
+
+##### DeepSeek
+
+DeepSeek does not have a dedicated image model, but the runtime
+generates SVG vector graphics through its text model. Each
+generation produces a valid SVG document that can be converted
+to PNG with tools like `rsvg-convert`. Pass an existing agent
+to maintain a session across generations:
+
+```ruby
+require "llm"
+llm = LLM.deepseek(key: ENV["KEY"])
+
+##
+# First generation
+res = llm.images.create(prompt: "a rocket on the moon")
+IO.copy_stream res.images[0], "rocket.svg"
+
+##
+# Refine with follow-up prompts (shares context)
+res = llm.images.create(prompt: "add a dog next to the rocket",
+                        agent: res.agent)
+IO.copy_stream res.images[0], "rocket-with-dog.svg"
+```
+
 ## FAQ
 
 <details>
@@ -437,6 +709,7 @@ In no particular order:
 🇺🇸 Anthropic <br>
 🇨🇳 DeepSeek <br>
 🇨🇳 zAI <br>
+🇨🇳 Moonshot AI (Kimi) <br>
 🇪🇺 Mistral <br>
 
 **Weights**
@@ -448,6 +721,7 @@ In no particular order:
 🇺🇸 AWS bedrock <br>
 🇨🇳 DeepSeek <br>
 🇨🇳 zAI <br>
+🇨🇳 Moonshot AI (Kimi) <br>
 🇪🇺 Mistral <br>
 
 **Local**
@@ -512,6 +786,41 @@ wasn't possible to cover every feature without the README becoming a small book.
 The [r.uby.dev](https://r.uby.dev) homepage also includes more learning material
 and resources.
 
+## Developers
+
+The llm.rb project is quite large and maintained primarily by one
+person. It would be near impossible for me to maintain both the codebase
+and its documentation, especially the [deepdive.md](https://r.uby.dev/llm/deepdive/)
+so I have written agents that maintain the documentation assets and that
+allows me to put more focus on the code.
+
+The following agents are available for those tasks, and all of them
+use the most cost effective option: DeepSeek. Feel free to use them
+in your own fork.
+
+```sh
+##
+# Maintains the deepdive and API docs
+rake agents:scribe:yardoc
+rake agents:scribe:coverage
+rake agents:scribe:regressions
+rake agents:scribe:style
+
+##
+# Maintains the release
+rake agents:dexter:changelog
+rake agents:dexter:release
+
+##
+# Maintains mruby-llm backports
+rake agents:mruby:research
+rake agents:mruby:implement
+
+##
+# Refresh the data/ registry
+rake models.dev:download
+```
+
 ## License
 
 This software is released under the terms of the MIT license. <br>

data/bin/llm.rb

@@ -1,7 +1,9 @@
 #!/usr/bin/env ruby
 
 require "llm"
+require "json"
 require "fileutils"
+require "securerandom"
 
 ##
 # utils
@@ -76,20 +78,18 @@ def main(argv)
       temp = true
     when '-p'
       provider = argv.shift
+      if provider.nil?
+        warn "llm.rb: -p switch requires an argument"
+        help
+        exit 1
+      end
     else
       warn "llm.rb: unknown option #{option}"
+      help
+      exit 1
     end
   end
 
-  ##
-  # Setup the home directory
-  # But only if the `-t` switch has not been provided
-  if temp.nil?
-    home = File.join(Dir.home, ".llm.rb")
-    FileUtils.mkdir_p(File.join(home, Dir.getwd))
-    session = File.join(home, Dir.getwd, "session.json")
-  end
-
   ##
   # No provider has been given.
   # Try to infer one.
@@ -102,19 +102,43 @@ def main(argv)
       provider, = key.split("_")
     end
   end
+  provider = provider.downcase
+
+  ##
+  # Setup the filesystem where <provider>.json maps
+  # the current working directory to a session file,
+  # and where the session file is stored in
+  # `~/.llm.rb/<provider>/<uuid>.json`.
+  # This can be skipped with the `-t` option.
+  if temp.nil?
+    home   = File.join(Dir.home, ".llm.rb")
+    file   = File.join(home, "#{provider}.json")
+    parent = File.join(home, provider)
+
+    FileUtils.mkdir_p(parent)
+    FileUtils.touch(file)
+
+    if File.size(file).zero?
+      data = LLM::Object.from({})
+      File.binwrite file, JSON.pretty_generate(data)
+    else
+      data = LLM::Object.from JSON.parse(File.read(file))
+    end
+    data[Dir.getwd] ||= File.join(parent, "#{SecureRandom.uuid}.json")
+  end
 
   ##
   # We're ready to start the REPL
   # This should always succeed unless -p gave garbage
-  provider = provider.downcase
   if LLM.respond_to?(provider)
     key ||= "#{provider.upcase}_API_KEY"
     if ENV[key].nil? || ENV[key].to_s.empty?
       warn "llm.rb: set #{key} to use #{provider}"
       exit 1
     end
-    llm = LLM.method(provider).call(key: ENV[key])
-    agent = LLM::Agent.new(llm, path: temp ? nil : session, tools: LLM::Tool.subclasses)
+    llm   = LLM.method(provider).call(key: ENV[key])
+    path  = temp ? nil : data[Dir.getwd]
+    agent = LLM::Agent.new(llm, path:, tools: LLM::Tool.subclasses)
     agent.repl
   else
     warn "llm.rb: #{provider} was not recognized"