llm.rb 13.0.0 → 14.0.0

data/resources/deepdive/advanced/context.md

@@ -0,0 +1,267 @@
+
+## Context
+
+### Introduction
+
+#### Overview
+
+[`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html)
+is the runtime that powers every agent. When you call
+[`LLM::Agent#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#talk),
+the agent delegates to its internal context. The context manages
+the message history, sends requests to the provider, tracks pending
+tool calls, and feeds results back to the model. Everything an agent
+does, a context does too, but without the automatic tool loop.
+
+Using a context directly gives you finer control over each step
+of the conversation. You decide when to send messages, when to
+execute tools, and when to stop. This is useful for custom
+confirmation flows, mixed concurrency strategies per tool, or
+any workflow where the agent's automatic loop gets in the way.
+
+#### How it works
+
+A context wraps a provider and maintains the conversation state.
+Call
+[`LLM::Context#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#talk)
+to send input to the model, check
+[`LLM::Context#pending_functions?`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#pending_functions?)
+to see if tools were requested, and use
+[`LLM::Context#wait`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#wait)
+to execute them. Each call to
+[`LLM::Context#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#talk)
+appends
+to the conversation and returns the model's response. The context
+serializes its state with
+[`LLM::Context#to_h`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#to_h)
+and
+[`LLM::Context#to_json`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#to_json),
+and restores it
+with
+[`LLM::Context#restore`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#restore).
+This is how the ORM integrations and filesystem
+persistence work under the hood:
+
+```ruby
+require "llm"
+
+llm = LLM.deepseek(key: ENV["KEY"])
+ctx = LLM::Context.new(llm)
+
+res = ctx.talk "What's the weather in Tokyo?"
+puts res.content
+```
+
+#### Why would I use it?
+
+A bare context gives you control that the agent
+abstraction does not expose. Pre-flight checks on tool requests,
+per-tool confirmation prompts, mixed concurrency strategies across
+tools, or manual iteration until a condition is met are all easier
+with a bare context.
+
+#### Notes
+
+The agent uses
+[`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html)
+internally. Anything you can do with
+a context, you can also do through an agent. The trade-off is
+convenience versus control. Contexts support the same concurrency
+strategies, compaction, cancellation, and serialization as agents.
+
+### Manual loop
+
+#### Overview
+
+[`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html)
+manages the tool loop automatically. It calls
+the model, checks for tool requests, runs the tools, feeds results
+back, and repeats until the model produces text. You can bypass
+this and drive
+[`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html)
+directly instead. This
+gives you finer control over each step of the loop at the cost of
+more code.
+
+#### How it works
+
+When you want to control the tool loop yourself, drive
+[`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html)
+directly instead of using an agent. Start a conversation, check
+for tool requests, execute them, and feed results back. The full
+loop is under your control. Each call to
+[`LLM::Context#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#talk)
+appends to the conversation and returns the model's response, and
+[`LLM::Context#pending_functions?`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#pending_functions?)
+tells you whether tools were requested.
+From that foundation you can inspect, iterate, or confirm
+per-tool in a single flow:
+
+```ruby
+require "llm"
+
+llm = LLM.deepseek(key: ENV["KEY"])
+ctx = LLM::Context.new(llm)
+
+loop do
+  res = ctx.talk("What's the weather in Tokyo?")
+  break unless ctx.pending_functions?
+
+  puts "Model requested #{ctx.pending_functions.size} tool(s)"
+
+  results = ctx.pending_functions.map do |fn|
+    print "Run #{fn.name} with #{fn.arguments}? [y/N] "
+    if $stdin.gets&.match?(/\Ay\z/i)
+      fn.task(:thread).wait
+    else
+      fn.cancel(reason: "user declined")
+    end
+  end
+
+  ctx.talk(results)
+end
+
+puts res.content
+```
+
+#### Why would I use it?
+
+Manual control gives you pre-execution checks, custom confirmation
+flows, different strategies per tool, and fine-grained error
+recovery that the default tool loop does not expose.
+
+#### Notes
+
+[`LLM::Context#wait`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#wait)
+picks up pending functions, spawns them using the chosen
+strategy, waits for results, and records them back in the context.
+Each strategy is supported: `:sequential`, `:thread`, `:fiber`,
+`:async`, `:fork`, and `:ractor`. Functions are reset after each
+[`LLM::Context#wait`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#wait)
+or
+[`LLM::Context#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#talk)
+call. Store the array if you need to
+preserve them.
+
+### Pending functions
+
+#### Overview
+
+Pending function calls represent the model's tool requests. After
+[`LLM::Context#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#talk)
+returns, the context may have pending function
+calls if the model requested tools. These are available through
+[`LLM::Context#pending_functions`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#pending_functions)
+which returns an array of
+[`LLM::Function`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html)
+objects. Each function has a name, arguments, and methods for
+execution or cancellation.
+
+#### How it works
+
+When you want to check whether the model requested tools, call
+[`LLM::Context#pending_functions?`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#pending_functions?)
+after each
+[`LLM::Context#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#talk)
+call. Each pending
+function has a name, arguments, and
+methods for execution or cancellation. Call
+[`LLM::Function#task`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html#task)
+to execute it or
+[`LLM::Function#cancel`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html#cancel)
+to skip it. Iterate over all
+pending functions to inspect or handle them individually:
+
+```ruby
+res = ctx.talk "What's the weather in Tokyo?"
+
+if ctx.pending_functions?
+  puts "Model requested #{ctx.pending_functions.size} tool(s)"
+  results = ctx.pending_functions.map do |fn|
+    print "Run #{fn.name} with #{fn.arguments}? [y/N] "
+    if $stdin.gets&.match?(/\Ay\z/i)
+      fn.task(:thread).wait
+    else
+      fn.cancel(reason: "user declined")
+    end
+  end
+  ctx.talk(results)
+end
+```
+
+#### Why would I use it?
+
+Inspecting pending functions lets you decide which tools to run,
+in what order, and with what strategy. This is essential for
+confirmation flows, selective execution, or logging which tools
+the model requested.
+
+#### Notes
+
+Pending functions are reset after each
+[`LLM::Context#wait`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#wait)
+or
+[`LLM::Context#talk`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#talk)
+call. If you need to preserve them, store the array before
+executing. Functions that are cancelled still count as completed
+from the model's perspective; the model sees a cancellation
+result, not a tool error.
+
+### Tool responses
+
+#### Overview
+
+A tool interrupt gives you two choices. When a tool receives
+[`LLM::Interrupt`](https://r.uby.dev/api-docs/llm.rb/LLM/Interrupt.html),
+it can either cancel the turn or return a result. The choice
+depends on the situation.
+A hard cancel aborts the request outright and is the default.
+Returning a value lets the model adapt and continue the
+conversation, which can be useful when the interrupt is
+temporary, like a timeout or a user pause.
+
+#### How it works
+
+When a tool receives
+[`LLM::Interrupt`](https://r.uby.dev/api-docs/llm.rb/LLM/Interrupt.html),
+re-raise to abort the turn or return a value to continue the loop.
+The model receives the result and decides what to do next.
+
+Re-raise to abort the turn entirely:
+
+```ruby
+class MyTool < LLM::Tool
+  def call
+    # do work
+  rescue LLM::Interrupt
+    cleanup
+    raise
+  end
+end
+```
+
+Return a value to continue the loop:
+
+```ruby
+class MyTool < LLM::Tool
+  def call
+    # do work
+  rescue LLM::Interrupt
+    cleanup
+    {ok: false, reason: "interrupted"}
+  end
+end
+```
+
+#### Why would I use it?
+
+A hard cancel aborts the request outright. Useful when continuing
+would produce garbage. Returning a value lets the model adapt,
+which can be helpful when the interrupt is temporary.
+
+#### Notes
+
+The mechanism is the same across all six concurrency strategies.
+The `:ractor` strategy delivers the interrupt through ractor
+message passing. The `:fork` strategy delivers it via xchan.
+

data/resources/deepdive/advanced/guard.md

@@ -0,0 +1,371 @@
+
+## Guard
+
+### Introduction
+
+#### Overview
+
+[`LLM::Guard`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard.html)
+is the superclass for context-level supervisors. A guard is bound
+to a context and inspects each pending tool call before it runs.
+It can let the call through, cancel it, block it with an error, or
+answer for it with a synthesized result. Beyond loop detection,
+guards handle policy, validation, quotas, cost control, caching,
+and approval workflows.
+
+#### How it works
+
+A guard is a subclass of
+[`LLM::Guard`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard.html)
+that implements
+[`LLM::Guard#call`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard.html#call-instance_method).
+The guard is stamped onto the functions the context binds, and when a
+function's task runs the guard is checked on the calling thread before
+the tool is handed to the strategy. `call` receives the pending
+`function:` and returns an
+[`LLM::Function::Return`](https://r.uby.dev/api-docs/llm.rb/LLM/Function/Return.html)
+to close that single call, or `nil` to let it run. The guard inspects
+the current conversation through the `messages` helper and reads the
+pending call through the function's own accessors.
+
+Configure a guard by passing a class through the `guard:` option;
+options given in `guard_options:` are forwarded to `call` as keyword
+arguments:
+
+```ruby
+class RateLimitGuard < LLM::Guard
+  def call(function:, limit: 10)
+    if messages.count(&:tool_return?) >= limit
+      function.return(error: true, type: "guard_error",
+                      message: "too many tool calls")
+    end
+  end
+end
+
+agent = LLM::Agent.new(
+  llm,
+  guard: RateLimitGuard,
+  guard_options: {limit: 3}
+)
+agent.talk "Research the market", tools: [FetchNews, FetchStocks]
+```
+
+#### Why would I use it?
+
+A guard is the one hook that sees every pending tool call and the
+full conversation at once. It can observe what the model is about
+to do, cancel a call that needs approval, block one that violates
+policy, answer a cheap question without running a tool, or stop
+work when a budget is spent. Because the guard runs before the
+tool, anything it intercepts never executes.
+
+#### Notes
+
+[`LLM::Guard::Null`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard/Null.html)
+is the default guard for
+[`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html);
+it never blocks tool work.
+[`LLM::Context#guard`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#guard)
+returns the configured guard class. Because `call` returns an
+[`LLM::Function::Return`](https://r.uby.dev/api-docs/llm.rb/LLM/Function/Return.html)
+rather than a warning string, a guard can block an individual tool
+call while the rest of the batch still executes. The guard is stamped
+onto every function the context binds, so it runs whenever a task is
+spawned — including tool calls a stream queues itself during a
+streaming turn. A blocked call yields its return without executing.
+
+The runtime binds a guard instance to the context and stamps it onto
+the functions it resolves, so a guard cannot carry state in instance
+variables between calls. Anything a guard needs to remember, like
+how many calls already ran, must come from the conversation
+(`messages`) or from class-level state.
+
+### Inspect
+
+#### Overview
+
+The guard receives the pending
+[`LLM::Function`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html)
+and can read what the model is about to do before anything runs.
+The function exposes its `name`, `arguments`, and parameter schema,
+so a guard can base its decision on the actual call rather than on
+the whole conversation. The guard also holds the context, so the
+message history, token usage, and cost are visible too.
+
+#### How it works
+
+When you want to inspect a pending call before it runs, read the
+function's accessors inside `call`.
+[`LLM::Function#name`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html#name)
+is the tool name, and
+[`LLM::Function#arguments`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html#arguments)
+is an
+[`LLM::Object`](https://r.uby.dev/api-docs/llm.rb/LLM/Object.html)
+with the parsed arguments. The full parameter schema is available
+through
+[`LLM::Function#params`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html#params).
+Returning `nil` lets the call run, so an inspection-only guard is a
+pure observer:
+
+```ruby
+class AuditGuard < LLM::Guard
+  def call(function:)
+    warn "pending: #{function.name}(#{function.arguments.inspect})"
+    nil  # let it run
+  end
+end
+```
+
+#### Why would I use it?
+
+Inspecting the pending call is the basis for policy, validation,
+and audit. A guard can log every call into a telemetry stream, or
+reject a single call because its arguments violate a rule while
+letting every other call through. The same read underpins the
+richer decisions in the rest of this document.
+
+#### Notes
+
+The guard sees the parsed arguments exactly as the model requested
+them. Reading them costs nothing and never executes the tool.
+Returning `nil` means the call proceeds normally.
+
+### Cancel
+
+#### Overview
+
+Cancelling is distinct from blocking with an error.
+[`LLM::Function#cancel`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html#cancel)
+produces a return the model reads as a declined call, not a failed
+one. The conversation stays honest: the model asked for something,
+and the runtime declined it with a reason.
+
+#### How it works
+
+When you want to cancel a single pending call, call the
+[`LLM::Function#cancel`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html#cancel)
+method with a `reason:` and return the result from `call`. The
+pending call never executes, and the rest of the batch still runs.
+A guard can cancel one call in a two-call batch and the other call
+still executes:
+
+```ruby
+class ApprovalGuard < LLM::Guard
+  def call(function:)
+    if function.name == "delete-file"
+      function.cancel(reason: "delete requires human approval")
+    end
+  end
+end
+```
+
+#### Why would I use it?
+
+Cancellation suits approval workflows and environment rules. A call
+that needs a human in the loop, a tool that is disabled for a
+particular user, or an operation that must not run in production
+can all be declined without pretending the tool failed. Because
+cancellation is per call, it declines only the offending tool and
+leaves the rest of the batch intact.
+
+#### Notes
+
+The model receives
+`{cancelled: true, reason: "delete requires human approval"}` as
+that call's result and can react, for example by asking for
+permission or skipping the operation. A cancelled call still counts
+as resolved, like any tool return, so the model keeps moving. The
+reason string is what the model sees, so write it as guidance
+("ask the user first") rather than a raw error dump.
+
+### Block
+
+#### Overview
+
+Blocking is how a guard enforces policy. A blocked call never runs,
+and the model receives an in-band error explaining why. Unlike a
+cancellation, an error tells the model the tool could not produce
+a result at all.
+
+#### How it works
+
+When you want to block a call, return an
+[`LLM::Function::Return`](https://r.uby.dev/api-docs/llm.rb/LLM/Function/Return.html)
+with `error: true`. The `type` and `message` are free-form, and the
+model sees them inside the return value. Returning `nil` lets the
+call through:
+
+```ruby
+class PolicyGuard < LLM::Guard
+  def call(function:)
+    if function.name == "shell"
+      function.return(error: true, type: "policy_error",
+                      message: "shell is disabled")
+    end
+  end
+end
+```
+
+#### Why would I use it?
+
+Blocking denies a dangerous tool, rejects an out-of-policy
+argument, or stops a quota violation before it happens. The model
+receives the return and can adapt, so the conversation stays valid.
+
+#### Notes
+
+A blocked call never executes. The return it produces is sent back
+through the model like any tool result, so the model sees why the
+call was blocked and can change course.
+
+### Answer
+
+#### Overview
+
+A guard's return is injected into the conversation as if the tool
+had executed, so a guard can answer for a tool that never runs.
+From the model's side, a synthesized result is indistinguishable
+from a real one.
+
+#### How it works
+
+When you want to answer a pending call yourself, return a value
+through the
+[`LLM::Function#return`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html)
+helper. The value you pass becomes the tool's result as-is, so it
+must look like a plausible answer:
+
+```ruby
+class CacheGuard < LLM::Guard
+  CACHE = {"get-weather:tokyo" => {forecast: "sunny"}}
+
+  def call(function:)
+    key = "#{function.name}:#{function.arguments[:city]}"
+    function.return(CACHE[key]) if CACHE.key?(key)
+  end
+end
+```
+
+#### Why would I use it?
+
+Standing in for a tool is how you cache expensive calls, mock tools
+in tests, or degrade gracefully when a service is down. The same
+hook can return a fixed answer for tools that should not run in a
+given environment.
+
+#### Notes
+
+[`LLM::Function#unavailable`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html)
+marks a tool as not found, and
+[`LLM::Function#budget_spent`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html)
+reports that the tool budget is exhausted. Both are returns too,
+so they flow back to the model like any tool result. A guard that
+synthesizes a result runs instead of the tool, so side effects the
+tool would have performed, like a database write, are skipped.
+
+### Budget
+
+#### Overview
+
+A guard can read the accumulated usage and cost of the conversation
+through the context. That makes it the natural place to enforce a
+hard ceiling: stop calling tools once a cost or token budget is
+spent, or once the context window is nearly full.
+
+#### How it works
+
+When you want to enforce a budget, compare
+[`LLM::Context#usage`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#usage)
+or
+[`LLM::Context#cost`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#cost-instance_method)
+against a limit inside `call`. Pass the limit through
+`guard_options:` so it can be tuned per context. Returning a
+cancellation or an error closes the call before it runs:
+
+```ruby
+class BudgetGuard < LLM::Guard
+  def call(function:, limit: 0.05)
+    if ctx.cost.total >= limit
+      function.cancel(reason: "cost ceiling reached, ask before continuing")
+    end
+  end
+end
+
+ctx = LLM::Context.new(
+  llm,
+  guard: BudgetGuard,
+  guard_options: {limit: 0.01}
+)
+```
+
+#### Why would I use it?
+
+Budgets matter for long autonomous runs where the model decides how
+many tools to call. A cost ceiling keeps a runaway agent from
+spending money, and a quota derived from `messages` (as in the
+rate-limit example above) keeps a single user from exhausting
+shared resources. The same comparison works with
+[`LLM::Context#usage`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#usage)
+against
+[`LLM::Context#context_window`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#context_window)
+to keep the conversation inside the model's window.
+
+#### Notes
+
+The cost reported by
+[`LLM::Context#cost`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html#cost-instance_method)
+reflects the conversation so far, so a limit is enforced per call:
+once the ceiling is crossed, the next pending call is declined. The
+guard is not a replacement for
+[`LLM::Agent.tool_budget`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#tool_budget-class_method),
+which caps the number of tool calls in a single turn. The two
+compose: the budget caps call count, and a guard enforces cost.
+
+### Loop
+
+#### Overview
+
+[`LLM::Guard::Loop`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard/Loop.html)
+is the built-in loop-detection guard. It reduces each assistant
+tool call to a `[tool name, arguments]` signature and checks whether
+the tail of the sequence is repeating.
+
+#### How it works
+
+When you want to detect repeated tool-call patterns, enable
+[`LLM::Guard::Loop`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard/Loop.html)
+and tune the `threshold:` option, which is the number of repeated
+patterns required before the guard intervenes (default `3`). When
+the guard detects a repeat, it returns an in-band
+[`LLM::Function::Return`](https://r.uby.dev/api-docs/llm.rb/LLM/Function/Return.html)
+with type `"guard_error"` and a message that tells the model it is
+stuck and should change approach:
+
+```ruby
+ctx = LLM::Context.new(
+  llm,
+  guard: LLM::Guard::Loop,
+  guard_options: {threshold: 2}
+)
+ctx.talk "Research the market", tools: [FetchNews, FetchStocks]
+```
+
+#### Why would I use it?
+
+Loop detection matters for long, autonomous agent runs. Without it,
+a model that repeats a tool call with the same arguments can
+bounce between calls forever. The guard turns that into a bounded
+conversation: after the threshold, the model receives a message
+telling it to stop and try a different strategy.
+
+#### Notes
+
+[`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html)
+enables
+[`LLM::Guard::Loop`](https://r.uby.dev/api-docs/llm.rb/LLM/Guard/Loop.html)
+by default, so agents get loop protection without configuration.
+A custom guard can be passed through the `guard:` option to replace
+the loop guard entirely. Guards and the agent's tool budget
+complement each other: a guard blocks work that looks stuck, while
+[`LLM::Agent.tool_budget`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#tool_budget-class_method)
+caps the total number of tool calls in a single turn.