patient_llm 0.8.0 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 55806cbf3beeef58969661d79224bf6ff527ac3e73d92e8426ef854c0635dca0
-  data.tar.gz: bdae58940e5c112a237d5e64455ed956d42577ad0d2241bb7ea0a4576df01f02
+  metadata.gz: 84d77f173d81a34af7c79a2a4052229916bf814660ef4a7616427601149bd8a9
+  data.tar.gz: 27f29411ac6e24fbb4c27da41f2681c0cd80576d5c6cda3ccf133761cf8ef92a
 SHA512:
-  metadata.gz: ca5d9f04997f0bf171a4de6f5edb599b2b33991c36ce4835cde1574e251b5e6d926887de53d5a31ff7d3e76b294bb9e56e41e467ebb671b3dc36d7031d128b7c
-  data.tar.gz: 533df1b81438b348387a5824df29b3a74a9102292f2cd82de910b0c7fe85d7864cd3364292e5bd5cdca93f10181026987926c681b349668efac6feab5fea9bd3
+  metadata.gz: a02bd347cfb0ff748157d25615a4a588b34ebd2c465ae62abef11dc50c2e3e347fa9c53986e2e1c549fa06353b6c9407a0faa978541bca17233ce3334d58f0d5
+  data.tar.gz: 8a7aa8f715eed1566716bc829b4a518d72c3dbc6e639d4564f9de40becbfceff36fbbf432b19ea2fea1a352a9f0304b6ca0e04f5e4fe1817e4f6aa1679a80d36

data/CHANGELOG.md

@@ -4,6 +4,35 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.10.0
+
+### Added
+
+- Streaming responses. Set `stream: true` on a provider, pass `stream:` to `ask` (on `PatientLLM` or an agent, including `continue` and `ask!`), or declare `stream true` on an agent. The provider streams the response so data keeps flowing during long generations, and the streamed body is reassembled into a single `PromptBuilder::Response` before the callback runs, so callbacks, hooks, and tool loops are unchanged. Every API format is supported; Gemini and Bedrock Converse use their streaming endpoints by default when streaming.
+- `read_timeout` provider and request option: the maximum number of seconds to wait for data on a streamed request. It is not applied to requests that are not streamed.
+- `max_response_size` provider and request option to raise the PatientHttp response size limit for LLM requests, since streamed bodies are larger than JSON bodies.
+- A stream that reports an error event, ends before its final event, or is malformed is sent to `on_error` (and an agent's `failed` hook) as a `PatientHttp::RequestError` with the error type `:stream_error`.
+
+### Changed
+
+- `stream:` passed to an agent's `ask` or `preview_request` is a request option instead of a session option, so it is accepted by `continue` and is not saved in the conversation state.
+- patient_http minimum version is now 1.8.0, which adds the `read_timeout` and per-request `max_response_size` options.
+- Minimum Ruby version is now 3.3.
+
+## 0.9.0
+
+### Added
+
+- `Agent.reasoning` accepts a block or callable argument like the other scalar declarations (e.g. `reasoning { LLMExperiment.reasoning_effort }`). It is evaluated each time a request is built, and its result is normalized with the same rules as a static value.
+- `reasoning false` turns reasoning off with `session.think(false)` (serialized as e.g. `thinking: {type: "disabled"}` on Anthropic Messages and Anthropic models on Converse), as opposed to `reasoning nil`, which sends no reasoning configuration and leaves the API's default in place. An effort level can accompany it (`reasoning false, effort: :low`). `reasoning true` turns reasoning on with the API's default depth.
+- `Agent.reasoning` also accepts a Hash of options (`{effort:}`, `{budget_tokens:}`, `{enabled: false}`), the form the reader returns.
+
+### Changed
+
+- The per-request `reasoning:` option to `ask`, `ask!`, and `preview_request` is normalized the same way as the `reasoning` declaration, so `reasoning: :low`, `reasoning: "low"`, `reasoning: {effort: :low}`, `reasoning: {budget_tokens: 2048}`, and `reasoning: false` produce the same session reasoning as the equivalent declaration. Previously the value was set on the session as-is, so shorthands such as `reasoning: :low` set the session reasoning to a String instead of a configuration Hash. `reasoning: nil` still sends no reasoning configuration for the request, and a Hash with other keys is still used as the raw reasoning configuration.
+- `Agent.reasoning` raises an `ArgumentError` for values that are not a reasoning configuration (such as a number) instead of converting them to an effort level string.
+- prompt_builder minimum version is now 0.5, which adds the explicit reasoning off configuration used by `reasoning false`.
+
 ## 0.8.0
 
 ### Added

data/README.md

@@ -149,7 +149,7 @@ end
 PatientHttp.register_secret("custom.api_key") { ENV["CUSTOM_API_KEY"] }
 ```
 
-Provider options: `url`, `headers`, `serializer`, `path`, `params` (merged into every payload), `preprocessors`, `timeout`, `max_tool_iterations`, `api_key` (see [provider presets](#provider-presets)), and `region` (used by region-templated presets such as `:bedrock_runtime`).
+Provider options: `url`, `headers`, `serializer`, `path`, `params` (merged into every payload), `preprocessors`, `timeout`, `stream`, `read_timeout`, `max_response_size` (see [streaming responses](#streaming-responses)), `max_tool_iterations`, `api_key` (see [provider presets](#provider-presets)), and `region` (used by region-templated presets such as `:bedrock_runtime`).
 
 Every provider option (except `name`, `preset`, and `api_key`) may also be a callable, evaluated each time the provider is looked up — i.e. on every request — so values can be generated dynamically at runtime (`api_key` is consumed at registration time to build the authentication header; a callable key is resolved on the processor side at dispatch):
 
@@ -173,6 +173,33 @@ PatientLLM.configure do |config|
 end
 ```
 
+### Streaming responses
+
+LLM requests that generate for a long time can fail without streaming: the connection sits idle while the model works, and proxies, NAT gateways, or the provider's own front end may drop it before the response arrives. Anthropic, for example, recommends streaming for any request that might take longer than 10 minutes. With `stream: true`, the provider sends the response as it is generated, so data keeps flowing for the whole request.
+
+The streamed body is reassembled into the same response the non-streaming endpoint would have returned before your callback runs (with prompt_builder's `Response.parse_stream`), so callbacks, hooks, tool loops, and structured output work the same way. Tokens are not delivered to your code as they arrive.
+
+```ruby
+PatientLLM.configure do |config|
+  config.provider :anthropic, preset: :anthropic, api_key: -> { ENV["ANTHROPIC_API_KEY"] },
+    stream: true,
+    timeout: 30 * 60,                    # limit for the full stream
+    read_timeout: 120,                   # limit for each wait for data
+    max_response_size: 16 * 1024 * 1024  # streamed bodies are larger than JSON bodies
+end
+```
+
+- `stream` turns streaming on for the provider's requests. A `stream:` option on `ask` (or an agent's `stream` declaration) overrides it for a request, including `stream: false`. A `stream` key in `params` is treated the same way: request settings win over provider settings, and at each level the `stream` option wins over a `stream` param. When none is set, the session's `stream` field applies.
+- `timeout` still limits the full request, so raise it for long generations.
+- `read_timeout` is the maximum number of seconds to wait for data from the server. The timer starts again each time a chunk arrives, so a stream that stops fails fast while a long stream that keeps sending data runs until `timeout`. It only applies to streamed requests: without streaming, the server sends nothing until it has generated the whole response.
+- `max_response_size` overrides the PatientHttp `max_response_size` (1MB by default) for the provider's requests. A streamed body is several times larger than the equivalent JSON body.
+
+Streaming works with every API format. The Chat Completions, Open Responses, and Messages formats send `stream: true` in the request body. Gemini and Bedrock Converse stream from a different endpoint, which is used by default when streaming (`v1beta/models/{model}:streamGenerateContent?alt=sse` and `model/{model}/converse-stream`); an explicit `path` is used as is.
+
+A stream that reports an error after the response started (such as an Anthropic `overloaded_error` event, which arrives with a 200 status), ends before its final event, or is malformed is sent to `on_error` (or an agent's `failed` hook) as a `PatientHttp::RequestError` with an `error_type` of `:stream_error`. The `error_class` names the prompt_builder error (`PromptBuilder::ErrorResponseError`, `PromptBuilder::IncompleteStreamError`, or `PromptBuilder::ParseError`), and `http_response` holds the streamed body.
+
+Streaming requires patient_http 1.8.0 or later.
+
 ### Request signing (preprocessors)
 
 Some providers require request signing rather than a static authentication header — for example, AWS Bedrock with SigV4, where a signature is computed over the final outgoing request. For these, register a [request preprocessor](https://github.com/bdurand/patient_http#request-preprocessors) on the PatientHttp configuration and reference it by name from the provider. Like secrets, only the preprocessor name is serialized into the job queue; the signing logic and credentials stay on the processor side.
@@ -213,17 +240,31 @@ class ResearchAgent < PatientLLM::Agent
   max_output_tokens 4_000
   reasoning :medium             # portable effort level, or reasoning budget_tokens: 8_000
   max_tool_iterations 5         # tool loop cap for this agent (default 10)
+  stream true                   # stream responses (see "Streaming responses")
   extra guardrail_config: {guardrailIdentifier: "gr-1", guardrailVersion: "1"}
 end
 ```
 
 `instructions` sets system-level instructions for the request. On APIs without a separate instructions field they are appended to the system prompt.
 
+`reasoning` configures reasoning (extended thinking), applied with prompt_builder's `session.think`. It accepts a portable effort level (`:minimal`, `:low`, `:medium`, `:high`, `:xhigh`, `:max`), `effort:` or `budget_tokens:`, or `false` to turn reasoning off. `nil` and `false` are different: `reasoning nil` means no reasoning configuration, so the request carries no reasoning parameter and the API's default applies (some models, such as recent Claude models, reason by default), while `reasoning false` explicitly turns reasoning off (e.g. `thinking: {type: "disabled"}` for Claude models). An effort level can accompany `false` (`reasoning false, effort: :low`) for APIs such as Anthropic Messages, where effort also limits token spend without thinking; APIs whose only reasoning control is effort ignore it and keep reasoning off.
+
+```ruby
+reasoning :medium                   # portable effort level
+reasoning budget_tokens: 8_000      # explicit thinking budget, where supported
+reasoning true                      # reasoning on, depth left to the API
+reasoning false                     # reasoning off
+reasoning false, effort: :low       # reasoning off, with an effort level on Messages
+reasoning nil                       # no reasoning configuration (removes an inherited value)
+```
+
+The reader returns the normalized `session.think` options (`{effort: "medium"}`, `{budget_tokens: 8000}`, `{}`, `{enabled: false}`, or `nil`).
+
 `extra` sets provider-specific extra data on every session the agent builds. The recognized keys depend on the serializer used for the request — for example, the Bedrock Converse serializer maps `guardrail_config`, `stop_sequences`, `additional_model_request_fields`, and `performance_config` into the request payload (see the prompt_builder docs for each serializer's keys). Redeclaring `extra` in a subclass replaces the whole hash, and `continue` re-applies the agent's current value to restored sessions. A per-request `ask(extra: {...})` replaces the declaration for that request; to add to it instead, merge explicitly: `MyAgent.ask(msg, extra: MyAgent.extra.merge(...))`. Session options such as `extra:` are only accepted when `ask` builds the session — passing them to `continue`, or together with a `session:`, raises `ArgumentError`.
 
 ### Dynamic values
 
-Every scalar declaration (`provider`, `model`, `system`, `instructions`, `temperature`, `max_output_tokens`, `max_tool_iterations`, and `extra`) also accepts a block in lieu of the argument, or a callable (anything responding to `call`) as the argument. The block or callable is evaluated each time the agent builds or sends a request, so the value can be generated dynamically at runtime:
+Every scalar declaration (`provider`, `model`, `system`, `instructions`, `temperature`, `max_output_tokens`, `reasoning`, `max_tool_iterations`, and `extra`) also accepts a block in lieu of the argument, or a callable (anything responding to `call`) as the argument. The block or callable is evaluated each time the agent builds or sends a request, so the value can be generated dynamically at runtime:
 
 ```ruby
 class ResearchAgent < PatientLLM::Agent
@@ -232,7 +273,21 @@ class ResearchAgent < PatientLLM::Agent
 end
 ```
 
-Validation and coercion (`extra`'s Hash check, `provider`'s symbol coercion) apply to the block's result each time it is evaluated.
+Validation and coercion (`extra`'s Hash check, `provider`'s symbol coercion, `reasoning`'s normalization) apply to the block's result each time it is evaluated. A `reasoning` block can return any value the static form accepts — an effort level, a Hash such as `{budget_tokens: 2048}`, `false` to turn reasoning off, or `nil` for no configuration:
+
+```ruby
+class SupportAgent < PatientLLM::Agent
+  model { LLMExperiment.model }
+  reasoning { LLMExperiment.reasoning_effort }   # e.g. :low, :high, or false
+  max_output_tokens { LLMExperiment.max_output_tokens }
+end
+```
+
+Per-request session options override the declarations for one request. The `reasoning:` option is normalized the same way as the declaration, so `reasoning: :low`, `reasoning: {budget_tokens: 2048}`, and `reasoning: false` produce the same request as the equivalent `reasoning` declaration, and `reasoning: nil` sends no reasoning configuration for that request. Any other Hash (e.g. `{summary: "auto"}`) is set as the session's raw reasoning configuration for provider-specific settings:
+
+```ruby
+SupportAgent.ask(message, reasoning: false, max_output_tokens: 1_000)
+```
 
 ### Inheritance
 
@@ -465,6 +520,9 @@ PatientLLM.ask(session,
   params: {max_completion_tokens: 1000},   # Additional request parameters
   preprocessors: :aws_sigv4,               # Replace the provider's request preprocessors
   timeout: 600,                            # Request timeout in seconds
+  stream: true,                            # Stream the response (see "Streaming responses")
+  read_timeout: 120,                       # Maximum wait for data on a streamed request
+  max_response_size: 16 * 1024 * 1024,     # Maximum response body size in bytes
   max_tool_iterations: 3,                  # Tool loop cap for this request
   processor: :llm                          # Named PatientHttp processor for this request
 )

data/VERSION

@@ -1 +1 @@
-0.8.0
+0.10.0

data/lib/patient_llm/agent.rb

@@ -68,6 +68,17 @@ module PatientLLM
     NOT_SPECIFIED = Object.new
     private_constant :NOT_SPECIFIED
 
+    # Keys accepted in a Hash reasoning configuration; they map to the
+    # arguments of `session.think`.
+    REASONING_OPTIONS = %i[enabled effort budget_tokens].freeze
+    private_constant :REASONING_OPTIONS
+
+    # Session options accepted by {ask} and {preview_request}. The session's
+    # stream field is excluded: stream: is a request option, so it can be
+    # used with {continue} and is not saved in the conversation state.
+    SESSION_OPTIONS = (PromptBuilder::Session::INITIALIZE_OPTIONS - [:stream]).freeze
+    private_constant :SESSION_OPTIONS
+
     class << self
       # DSL: get or set the provider name for this agent. The value is
       # inherited from the parent agent class unless set; pass an explicit nil
@@ -131,30 +142,48 @@ module PatientLLM
         get_or_set_setting(:temperature, value, block)
       end
 
-      # DSL: get or set the reasoning configuration. Accepts a portable effort
-      # level (:minimal, :low, :medium, :high, :xhigh, :max) or explicit options
-      # (effort: or budget_tokens:) which are applied with `session.think`.
-      # The value is inherited from the parent agent class unless set; pass an
-      # explicit nil to remove an inherited value.
+      # DSL: get or set the reasoning configuration, applied to sessions with
+      # `session.think`. Accepts a portable effort level (:minimal, :low,
+      # :medium, :high, :xhigh, :max), explicit options (effort: or
+      # budget_tokens:), or false to turn reasoning off; an effort: may
+      # accompany false for APIs that limit token spend with it while
+      # thinking is off. True turns reasoning on and leaves the depth to the
+      # API. A Hash of the same options (:effort, :budget_tokens, :enabled) is
+      # also accepted, which is the form the reader returns.
+      #
+      # Nil and false are different: nil means no reasoning configuration,
+      # so the request carries no reasoning parameter and the API's default
+      # applies (some models reason by default), while false explicitly turns
+      # reasoning off. The value is inherited from the parent agent class
+      # unless set; pass an explicit nil to remove an inherited value.
+      #
+      # @example
+      #   reasoning :medium
+      #   reasoning budget_tokens: 8_000
+      #   reasoning false
+      #   reasoning { ReasoningSettings.effort }  # e.g. :low, :high, or false
       #
-      # @param value [Symbol, String, nil] a portable effort level.
+      # @param value [Symbol, String, Hash, Boolean, nil] a portable effort
+      #   level, a Hash of options, or true/false to turn reasoning on or off.
       #   Passing an explicit nil removes the previously set value.
       # @param effort [Symbol, String, nil] explicit effort level
       # @param budget_tokens [Integer, nil] explicit thinking token budget
-      # @return [Hash, nil] the reasoning options
-      def reasoning(value = NOT_SPECIFIED, effort: nil, budget_tokens: nil)
-        return inherited_setting(:reasoning) if value.equal?(NOT_SPECIFIED) && effort.nil? && budget_tokens.nil?
-
-        passed = [(value.equal?(NOT_SPECIFIED) ? nil : value), effort, budget_tokens].compact
-        raise ArgumentError, "pass a level, effort:, or budget_tokens: — not more than one" if passed.size > 1
-
-        @reasoning = if passed.empty?
-          nil
-        elsif effort || budget_tokens
-          {effort: effort&.to_s, budget_tokens: budget_tokens}.compact
-        else
-          {effort: value.to_s}
+      # @yield an optional block (or callable argument) called each time the
+      #   value is read so it can be generated dynamically; the result accepts
+      #   the same forms as +value+ and is normalized at read time.
+      # @return [Hash, nil] the `session.think` options: +{effort:}+,
+      #   +{budget_tokens:}+, +{}+ for API defaults, or +{enabled: false}+ (with an
+      #   optional effort:) for reasoning off; nil for no configuration
+      # @raise [ArgumentError] if more than one of a level, effort:, or
+      #   budget_tokens: is passed, or budget_tokens: is passed with false
+      def reasoning(value = NOT_SPECIFIED, effort: nil, budget_tokens: nil, &block)
+        if effort || budget_tokens
+          raise ArgumentError, "pass either an argument or a block, not both" if block
+
+          value = normalize_reasoning(value.equal?(NOT_SPECIFIED) ? nil : value, effort: effort, budget_tokens: budget_tokens)
         end
+
+        get_or_set_setting(:reasoning, value, block) { |resolved| normalize_reasoning(resolved) }
       end
 
       # DSL: get or set the maximum output tokens. The value is inherited from
@@ -181,6 +210,22 @@ module PatientLLM
         get_or_set_setting(:max_tool_iterations, value, block)
       end
 
+      # DSL: get or set whether this agent's requests stream their responses.
+      # The streamed body is reassembled into a single response before the
+      # hooks run, so streaming only changes how the response travels: data
+      # keeps flowing during long generations instead of the connection
+      # sitting idle. The value is inherited from the parent agent class
+      # unless set; pass an explicit nil to remove an inherited value and use
+      # the provider's setting. A stream: option passed to {ask} overrides it.
+      #
+      # @param value [Boolean, nil] whether to stream. Passing an explicit nil removes the previously set value.
+      # @yield an optional block (or callable argument) called each time the
+      #   value is read so it can be generated dynamically.
+      # @return [Boolean, nil]
+      def stream(value = NOT_SPECIFIED, &block)
+        get_or_set_setting(:stream, value, block)
+      end
+
       # DSL: get or set provider-specific extra data set on sessions built by
       # this agent (e.g. the Bedrock Converse :guardrail_config). The
       # recognized keys depend on the serializer used for the request. The
@@ -276,19 +321,23 @@ module PatientLLM
       #   agent. Defaults to sending the hooks to the agent itself.
       # @param options [Hash] per-request overrides forwarded to {PatientLLM.ask}
       #   (url:, serializer:, path:, headers:, params:, preprocessors:, timeout:,
-      #   max_tool_iterations:, processor:), plus session options (model:, temperature:,
+      #   stream:, read_timeout:, max_response_size:, max_tool_iterations:,
+      #   processor:), plus session options (model:, temperature:,
       #   extra:, etc. — any PromptBuilder::Session::INITIALIZE_OPTIONS key)
       #   applied to the newly built session; session options raise
-      #   ArgumentError when a session: is supplied.
+      #   ArgumentError when a session: is supplied. The reasoning: option
+      #   accepts the same values as the {reasoning} DSL (e.g.
+      #   +reasoning: :low+ or +reasoning: false+) and is normalized the same
+      #   way; see {build_session}.
       # @return [Object] handler-specific identifier for the enqueued request
       def ask(message = nil, context: {}, session: nil, callback: nil, **options)
-        session_options = options.slice(*PromptBuilder::Session::INITIALIZE_OPTIONS)
+        session_options = options.slice(*SESSION_OPTIONS)
         raise ArgumentError.new("session options cannot be passed when a session is provided") if session && session_options.any?
 
         session ||= build_session(**session_options)
         session.user(message) if message
 
-        ask_options = options.except(*PromptBuilder::Session::INITIALIZE_OPTIONS)
+        ask_options = with_declared_stream(options.except(*SESSION_OPTIONS))
         iterations = max_tool_iterations
         ask_options[:max_tool_iterations] ||= iterations if iterations
 
@@ -371,7 +420,7 @@ module PatientLLM
       # @param options [Hash] the same per-request overrides accepted by {ask}
       # @return [PatientLLM::RequestPreview] the url, headers, and JSON payload
       def preview_request(message = nil, session: nil, context: nil, callback: nil, **options)
-        session_options = options.slice(*PromptBuilder::Session::INITIALIZE_OPTIONS)
+        session_options = options.slice(*SESSION_OPTIONS)
         raise ArgumentError.new("session options cannot be passed when a session is provided") if session && session_options.any?
 
         session ||= build_session(**session_options)
@@ -380,7 +429,7 @@ module PatientLLM
         PatientLLM.preview_request(
           session,
           provider: provider_name!,
-          **options.except(*PromptBuilder::Session::INITIALIZE_OPTIONS)
+          **with_declared_stream(options.except(*SESSION_OPTIONS))
         )
       end
 
@@ -388,10 +437,27 @@ module PatientLLM
       # the caller take precedence over the agent's declarations for the same
       # fields (pass an explicit nil to unset a declared value for one request).
       #
+      # The reasoning: option accepts the same values as the {reasoning} DSL
+      # and is normalized the same way, so +reasoning: :low+,
+      # +reasoning: {budget_tokens: 2048}+, and +reasoning: false+ produce the
+      # same session reasoning as the equivalent declarations. An explicit
+      # +reasoning: nil+ sends no reasoning configuration for the request. A
+      # Hash with keys other than :effort, :budget_tokens, and :enabled is
+      # used as the session's raw reasoning configuration for provider-specific
+      # settings.
+      #
+      # @param options [Hash] PromptBuilder::Session::INITIALIZE_OPTIONS values
       # @return [PromptBuilder::Session]
+      # @raise [ArgumentError] if the reasoning: option is invalid
       def build_session(**options)
+        overridden = options.keys
+        if options.key?(:reasoning) && !raw_reasoning_configuration?(options[:reasoning])
+          request_reasoning = normalize_reasoning(options.delete(:reasoning))
+        end
+
         session = PromptBuilder::Session.new(**options)
-        apply_configuration(session, except: options.keys)
+        apply_reasoning(session, request_reasoning) if request_reasoning
+        apply_configuration(session, except: overridden)
         session
       end
 
@@ -410,13 +476,14 @@ module PatientLLM
         temperature_value = temperature
         max_output_tokens_value = max_output_tokens
         extra_value = extra
+        reasoning_value = reasoning
         apply_system_message(session, system_message) if system_message && !except.include?(:system)
         session.model = model_value if model_value && !except.include?(:model)
         session.instructions = instructions_value if instructions_value && !except.include?(:instructions)
         session.temperature = temperature_value if temperature_value && !except.include?(:temperature)
         session.max_output_tokens = max_output_tokens_value if max_output_tokens_value && !except.include?(:max_output_tokens)
         session.extra = extra_value if extra_value && !except.include?(:extra)
-        session.think(**reasoning.transform_keys(&:to_sym)) if reasoning && !except.include?(:reasoning)
+        apply_reasoning(session, reasoning_value) if reasoning_value && !except.include?(:reasoning)
         if output_schema && !except.include?(:text)
           session.json_output(output_schema[:schema], name: output_schema[:name], strict: output_schema[:strict])
         end
@@ -445,6 +512,15 @@ module PatientLLM
         provider || raise(ArgumentError, "#{self} must declare a provider")
       end
 
+      # Add the agent's stream declaration to request options that do not set
+      # stream: themselves.
+      def with_declared_stream(options)
+        return options if options.key?(:stream)
+
+        declared = stream
+        declared.nil? ? options : options.merge(stream: declared)
+      end
+
       # Resolve the :callback option to the class name that travels through the
       # job queue. The class is resolved here so an unusable callback fails
       # where ask was called rather than later in a worker process.
@@ -485,6 +561,58 @@ module PatientLLM
         end
       end
 
+      # Normalize a reasoning value into the `session.think` options returned
+      # by {reasoning}: nil for no configuration, +{enabled: false}+ (with an
+      # optional effort) for reasoning off, and +{effort:}+, +{budget_tokens:}+,
+      # or +{}+ (API defaults) for reasoning on. Already-normalized values pass
+      # through unchanged.
+      def normalize_reasoning(value, effort: nil, budget_tokens: nil)
+        case value
+        when nil
+          return nil if effort.nil? && budget_tokens.nil?
+
+          normalize_reasoning(true, effort: effort, budget_tokens: budget_tokens)
+        when true
+          raise ArgumentError, "pass a level, effort:, or budget_tokens: — not more than one" if effort && budget_tokens
+
+          {effort: effort&.to_s, budget_tokens: budget_tokens}.compact
+        when false
+          raise ArgumentError, "budget_tokens: cannot be combined with reasoning false" if budget_tokens
+
+          {enabled: false, effort: effort&.to_s}.compact
+        when Symbol, String
+          raise ArgumentError, "pass a level, effort:, or budget_tokens: — not more than one" if effort || budget_tokens
+
+          {effort: value.to_s}
+        when Hash
+          raise ArgumentError, "pass reasoning options as a Hash or as keywords, not both" if effort || budget_tokens
+
+          options = value.transform_keys(&:to_sym)
+          unknown = options.keys - REASONING_OPTIONS
+          raise ArgumentError, "unknown reasoning options: #{unknown.join(", ")}" unless unknown.empty?
+
+          enabled = options.fetch(:enabled, true)
+          raise ArgumentError, "reasoning enabled: must be true or false" unless enabled == true || enabled == false
+
+          normalize_reasoning(enabled, effort: options[:effort], budget_tokens: options[:budget_tokens])
+        else
+          raise ArgumentError, "reasoning must be an effort level, a Hash of options, true, false, or nil; got #{value.inspect}"
+        end
+      end
+
+      # Whether a per-request reasoning value is a provider-specific Hash to
+      # set on the session as-is rather than options to normalize.
+      def raw_reasoning_configuration?(value)
+        value.is_a?(Hash) && value.keys.any? { |key| !REASONING_OPTIONS.include?(key.to_sym) }
+      end
+
+      # Apply normalized reasoning options (see {normalize_reasoning}) to a
+      # session with `session.think`.
+      def apply_reasoning(session, options)
+        options = options.dup
+        session.think(options.delete(:enabled) != false, **options)
+      end
+
       # Shared implementation of the scalar get-or-set DSL methods. With no
       # argument and no block, reads the setting; when the stored value
       # responds to #call it is called and the result used, so values can be

data/lib/patient_llm/callback.rb

@@ -17,6 +17,12 @@ module PatientLLM
   # The user callback receives a `PromptBuilder::Response` object. Access
   # the response text via `response.text`, token usage via `response.usage`,
   # and model id via `response.model`.
+  #
+  # A streamed response body is reassembled into the same response the
+  # non-streaming endpoint would have returned. A stream that reports an error
+  # event, ends before its terminal event, or is malformed is routed to the
+  # user callback's on_error method as a `PatientHttp::RequestError` with the
+  # error type :stream_error.
   class Callback
     # Default maximum number of tool-execution rounds before the loop stops and
     # routes a `PatientHttp::RequestError` (error type :max_tool_iterations) to
@@ -87,9 +93,21 @@ module PatientLLM
       prepare_user_callback(user_callback, session: session, provider: provider_name, callback_args: callback_args, http_response: response, request_id: original_request_id)
 
       serializer = resolve_serializer(callback_args)
-      # The headers let formats that return response metadata in HTTP headers
-      # (e.g. the Converse response id) populate the parsed response.
-      llm_response = PromptBuilder::Response.parse(response.json, serializer, headers: response.headers)
+      if streamed_response?(response, callback_args)
+        begin
+          # A response with no body (such as a 204) is an empty stream, which
+          # the parser reports as incomplete.
+          llm_response = PromptBuilder::Response.parse_stream(response.body || "", serializer, headers: response.headers)
+        rescue PromptBuilder::UnexpectedPayloadError, PromptBuilder::ParseError => e
+          error = request_error(e, :stream_error, response, original_request_id)
+          invoke_user_callback(user_callback, :on_error, session: session, provider: provider_name, callback_args: user_callback_args(callback_args), error: error, http_response: response, request_id: original_request_id)
+          return
+        end
+      else
+        # The headers let formats that return response metadata in HTTP headers
+        # (e.g. the Converse response id) populate the parsed response.
+        llm_response = PromptBuilder::Response.parse(response.json, serializer, headers: response.headers)
+      end
 
       if should_auto_execute_tools?(llm_response, user_callback)
         continue_tool_loop(session, provider_name, llm_response, callback_args, response, user_callback, original_request_id)
@@ -192,6 +210,18 @@ module PatientLLM
       provider_config&.dig(:serializer) || :chat_completion
     end
 
+    # Whether the response body is a stream. The stream flag recorded at
+    # dispatch covers requests streamed by request option, stream param,
+    # provider setting, or session field; the content type also catches an
+    # endpoint that streams without being asked, such as a custom path to a
+    # streaming endpoint.
+    def streamed_response?(response, callback_args)
+      return true if callback_args.fetch(:stream, nil) == true
+
+      content_type = response.headers["content-type"].to_s.downcase
+      STREAM_CONTENT_TYPES.any? { |type| content_type.start_with?(type) }
+    end
+
     def max_tool_iterations(callback_args)
       callback_args.fetch(:max_tool_iterations, MAX_TOOL_ITERATIONS).to_i
     end
@@ -274,11 +304,17 @@ module PatientLLM
     def max_tool_iterations_error(callback_args, http_response, request_id)
       limit = max_tool_iterations(callback_args)
       exception = MaxToolIterationsError.new("Tool-call loop exceeded #{limit} iterations")
+      request_error(exception, :max_tool_iterations, http_response, request_id)
+    end
+
+    # Wrap an exception raised while handling a response in a RequestError so
+    # it reaches the user callback's on_error method like a transport error.
+    def request_error(exception, error_type, http_response, request_id)
       PatientHttp::RequestError.new(
         class_name: exception.class.name,
         message: exception.message,
-        backtrace: [],
-        error_type: :max_tool_iterations,
+        backtrace: exception.backtrace || [],
+        error_type: error_type,
         duration: http_response&.duration,
         request_id: request_id,
         url: http_response&.url,

data/lib/patient_llm/configuration.rb

@@ -69,11 +69,18 @@ module PatientLLM
     # @param preprocessors [String, Symbol, Array<String, Symbol>, #call, nil] Names of request
     #   preprocessors to apply to every request (e.g. for request signing). Names must be
     #   registered on the PatientHttp configuration with `register_preprocessor`.
-    # @param timeout [Numeric, #call, nil] Request timeout in seconds for this provider's requests
+    # @param timeout [Numeric, #call, nil] Request timeout in seconds for this provider's requests.
+    #   For a streamed request this limits the full stream.
+    # @param stream [Boolean, #call, nil] Whether to stream this provider's responses. The
+    #   streamed body is reassembled into a single response before the callback runs.
+    # @param read_timeout [Numeric, #call, nil] Maximum seconds to wait for data from the
+    #   server on a streamed request. Not applied to requests that are not streamed.
+    # @param max_response_size [Integer, #call, nil] Maximum response body size in bytes
+    #   for this provider's requests. A streamed body is larger than the equivalent JSON body.
     # @param max_tool_iterations [Integer, #call, nil] Maximum automatic tool-execution rounds
     #   for this provider's requests (default: PatientLLM::Callback::MAX_TOOL_ITERATIONS)
     # @return [void]
-    def provider(name, preset: nil, url: nil, api_key: nil, region: nil, headers: nil, serializer: nil, path: nil, params: {}, preprocessors: nil, timeout: nil, max_tool_iterations: nil)
+    def provider(name, preset: nil, url: nil, api_key: nil, region: nil, headers: nil, serializer: nil, path: nil, params: {}, preprocessors: nil, timeout: nil, stream: nil, read_timeout: nil, max_response_size: nil, max_tool_iterations: nil)
       preset_config = preset ? Presets.fetch(preset) : nil
       options = {
         preset_config: preset_config,
@@ -85,6 +92,9 @@ module PatientLLM
         params: params,
         preprocessors: preprocessors,
         timeout: timeout,
+        stream: stream,
+        read_timeout: read_timeout,
+        max_response_size: max_response_size,
         max_tool_iterations: max_tool_iterations
       }
 
@@ -167,6 +177,9 @@ module PatientLLM
         params: options[:params] || {},
         preprocessors: options[:preprocessors],
         timeout: options[:timeout],
+        stream: options[:stream],
+        read_timeout: options[:read_timeout],
+        max_response_size: options[:max_response_size],
         max_tool_iterations: options[:max_tool_iterations]
       }
     end

data/lib/patient_llm.rb

@@ -31,6 +31,18 @@ module PatientLLM
     gemini: "v1beta/models/{model}:generateContent"
   }.freeze
 
+  # Default API paths per serializer format for streamed requests. Gemini and
+  # Bedrock Converse select streaming by endpoint; the other formats stream
+  # from the same endpoint with `stream: true` in the request body.
+  STREAMING_SERIALIZER_PATHS = SERIALIZER_PATHS.merge(
+    converse: "model/{model}/converse-stream",
+    gemini: "v1beta/models/{model}:streamGenerateContent?alt=sse"
+  ).freeze
+
+  # Content types of streamed response bodies: Server-Sent Events, and the AWS
+  # event-stream framing used by Bedrock Converse.
+  STREAM_CONTENT_TYPES = ["text/event-stream", "application/vnd.amazon.eventstream"].freeze
+
   # Required version header for the Anthropic Messages API.
   ANTHROPIC_VERSION = "2023-06-01"
 
@@ -153,6 +165,15 @@ module PatientLLM
     #   preprocessors; pass an empty array to clear them for this request.
     # @param timeout [Numeric, nil] Request timeout in seconds for this request. Overrides
     #   the provider's timeout; defaults to the PatientHttp processor configuration.
+    #   For a streamed request this limits the full stream.
+    # @param stream [Boolean, nil] Whether to stream the response. The streamed body is
+    #   reassembled into a single response before the callback runs. Overrides the
+    #   provider's setting; when neither is set, the session's `stream` field applies.
+    # @param read_timeout [Numeric, nil] Maximum seconds to wait for data from the
+    #   server on a streamed request. Overrides the provider's read timeout. Not
+    #   applied to requests that are not streamed.
+    # @param max_response_size [Integer, nil] Maximum response body size in bytes.
+    #   Overrides the provider's setting; defaults to the PatientHttp configuration.
     # @param max_tool_iterations [Integer, nil] Maximum automatic tool-execution rounds
     #   for this request. Overrides the provider's setting; defaults to
     #   {Callback::MAX_TOOL_ITERATIONS}.
@@ -160,7 +181,7 @@ module PatientLLM
     #   should execute this request. Overrides the configuration's processor;
     #   defaults to nil, which uses the default processor.
     # @return [Object] Handler-specific identifier for the enqueued request
-    def ask(session, provider:, callback:, callback_args: {}, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, max_tool_iterations: nil, processor: nil)
+    def ask(session, provider:, callback:, callback_args: {}, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, stream: nil, read_timeout: nil, max_response_size: nil, max_tool_iterations: nil, processor: nil)
       request_options = build_request_options(
         url: url,
         serializer: serializer,
@@ -169,6 +190,9 @@ module PatientLLM
         params: params,
         preprocessors: preprocessors,
         timeout: timeout,
+        stream: stream,
+        read_timeout: read_timeout,
+        max_response_size: max_response_size,
         max_tool_iterations: max_tool_iterations,
         processor: processor
       )
@@ -197,12 +221,18 @@ module PatientLLM
     # @param preprocessors [String, Symbol, Array<String, Symbol>, nil] Accepted for
     #   parity with {.ask}; preprocessors are not applied to the preview.
     # @param timeout [Numeric, nil] Accepted for parity with {.ask}; does not affect the preview.
+    # @param stream [Boolean, nil] Whether to stream the response. Streaming changes the
+    #   endpoint path for some formats and the payload, so it is reflected in the preview.
+    # @param read_timeout [Numeric, nil] Accepted for parity with {.ask}; does not affect
+    #   the preview.
+    # @param max_response_size [Integer, nil] Accepted for parity with {.ask}; does not
+    #   affect the preview.
     # @param max_tool_iterations [Integer, nil] Accepted for parity with {.ask}; does not
     #   affect the preview.
     # @param processor [String, Symbol, nil] Accepted for parity with {.ask}; does not
     #   affect the preview.
     # @return [RequestPreview] The url, headers, and JSON payload the request would send
-    def preview_request(session, provider:, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, max_tool_iterations: nil, processor: nil)
+    def preview_request(session, provider:, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, stream: nil, read_timeout: nil, max_response_size: nil, max_tool_iterations: nil, processor: nil)
       request_options = build_request_options(
         url: url,
         serializer: serializer,
@@ -211,6 +241,9 @@ module PatientLLM
         params: params,
         preprocessors: preprocessors,
         timeout: timeout,
+        stream: stream,
+        read_timeout: read_timeout,
+        max_response_size: max_response_size,
         max_tool_iterations: max_tool_iterations,
         processor: processor
       )
@@ -245,6 +278,7 @@ module PatientLLM
         callback: callback.to_s,
         custom: PromptBuilder.jsonify(callback_args || {}),
         request_options: request_options,
+        stream: resolved.stream,
         max_tool_iterations: resolved.max_tool_iterations,
         tool_iteration: tool_iteration,
         original_request_id: original_request_id
@@ -258,6 +292,8 @@ module PatientLLM
           headers: resolved.headers,
           preprocessors: resolved.preprocessors,
           timeout: resolved.timeout,
+          read_timeout: resolved.read_timeout,
+          max_response_size: resolved.max_response_size,
           processor: resolved.processor
         )
         PatientHttp.execute_inline(
@@ -273,6 +309,8 @@ module PatientLLM
           headers: resolved.headers,
           preprocessors: resolved.preprocessors,
           timeout: resolved.timeout,
+          read_timeout: resolved.read_timeout,
+          max_response_size: resolved.max_response_size,
           processor: resolved.processor,
           raise_error_responses: true,
           callback: PatientLLM::Callback,
@@ -285,14 +323,14 @@ module PatientLLM
 
     # Fully resolved request produced by merging per-request options over the
     # provider configuration.
-    ResolvedRequest = Data.define(:url, :serializer, :headers, :payload, :preprocessors, :timeout, :max_tool_iterations, :processor)
+    ResolvedRequest = Data.define(:url, :serializer, :headers, :payload, :preprocessors, :timeout, :stream, :read_timeout, :max_response_size, :max_tool_iterations, :processor)
     private_constant :ResolvedRequest
 
     # Normalize per-request overrides into a request options hash. The request
     # options travel through the job queue in the callback args, which only
     # permit JSON-native values; convert Symbols (e.g. serializer names,
     # preprocessor names, header/param values) to Strings up front.
-    def build_request_options(url:, serializer:, path:, headers:, params:, preprocessors:, timeout:, max_tool_iterations:, processor:)
+    def build_request_options(url:, serializer:, path:, headers:, params:, preprocessors:, timeout:, stream:, read_timeout:, max_response_size:, max_tool_iterations:, processor:)
       request_options = {}
       request_options["url"] = url if url
       request_options["serializer"] = serializer.to_s if serializer
@@ -301,6 +339,10 @@ module PatientLLM
       request_options["params"] = params if params && !params.empty?
       request_options["preprocessors"] = preprocessors if preprocessors
       request_options["timeout"] = timeout if timeout
+      # An explicit false is kept so a request can turn off a provider's streaming.
+      request_options["stream"] = stream unless stream.nil?
+      request_options["read_timeout"] = read_timeout if read_timeout
+      request_options["max_response_size"] = max_response_size if max_response_size
       request_options["max_tool_iterations"] = max_tool_iterations if max_tool_iterations
       request_options["processor"] = processor.to_s if processor
 
@@ -318,7 +360,11 @@ module PatientLLM
       resolved_serializer = (request_options["serializer"] || provider_config[:serializer] || :chat_completion).to_sym
       validate_serializer!(resolved_serializer)
 
-      resolved_path = request_options["path"] || provider_config[:path] || SERIALIZER_PATHS[resolved_serializer]
+      provider_params = deep_stringify_keys(provider_config[:params])
+      request_params = deep_stringify_keys(request_options["params"])
+      stream = resolve_stream(session, provider_config, request_options, provider_params, request_params)
+      default_paths = stream ? STREAMING_SERIALIZER_PATHS : SERIALIZER_PATHS
+      resolved_path = request_options["path"] || provider_config[:path] || default_paths[resolved_serializer]
       if resolved_path.include?("{model}")
         raise ArgumentError, "The endpoint path #{resolved_path.inspect} includes a {model} placeholder but session.model is not set" if session.model.nil?
 
@@ -333,10 +379,13 @@ module PatientLLM
         resolved_headers = {"anthropic-version" => ANTHROPIC_VERSION}.merge(resolved_headers)
       end
 
-      resolved_params = (provider_config[:params] || {}).merge(request_options["params"] || {})
+      # The stream param is resolved with the stream option above and set
+      # through the serializer, which adds the options streaming needs (such
+      # as the Chat Completions usage option), so it is not merged on top.
+      resolved_params = provider_params.merge(request_params).except("stream")
 
-      payload = session.request_payload(resolved_serializer)
-      payload = deep_merge(payload, deep_stringify_keys(resolved_params)) unless resolved_params.empty?
+      payload = streaming_payload(session, resolved_serializer, stream)
+      payload = deep_merge(payload, resolved_params) unless resolved_params.empty?
 
       ResolvedRequest.new(
         url: join_url(resolved_url, resolved_path),
@@ -345,11 +394,45 @@ module PatientLLM
         payload: payload,
         preprocessors: request_options["preprocessors"] || provider_config[:preprocessors],
         timeout: request_options["timeout"] || provider_config[:timeout],
+        stream: stream,
+        # The read timeout covers the wait for the response headers, which a
+        # non-streamed LLM request spends generating the whole response, so it
+        # only applies to streamed requests.
+        read_timeout: stream ? (request_options["read_timeout"] || provider_config[:read_timeout]) : nil,
+        max_response_size: request_options["max_response_size"] || provider_config[:max_response_size],
         max_tool_iterations: (request_options["max_tool_iterations"] || provider_config[:max_tool_iterations] || Callback::MAX_TOOL_ITERATIONS).to_i,
         processor: request_options["processor"] || configured_processor
       )
     end
 
+    # Whether the request streams its response. The first value that is set
+    # wins: the request option, a stream param on the request, the provider
+    # setting, a stream param on the provider, then the session's stream field.
+    def resolve_stream(session, provider_config, request_options, provider_params, request_params)
+      values = [
+        request_options["stream"],
+        request_params["stream"],
+        provider_config[:stream],
+        provider_params["stream"],
+        session.stream
+      ]
+      values.find { |value| !value.nil? } == true
+    end
+
+    # Serialize the session with its stream field set to match the resolved
+    # streaming setting. The field is set on a shallow copy, so the setting
+    # does not become part of the conversation state and concurrent requests
+    # sharing the session do not see each other's setting. A session that never
+    # set the field keeps it unset when the request is not streamed.
+    def streaming_payload(session, serializer, stream)
+      value = stream || (session.stream.nil? ? nil : false)
+      return session.request_payload(serializer) if value == session.stream
+
+      copy = session.dup
+      copy.stream = value
+      copy.request_payload(serializer)
+    end
+
     # The processor name from the configuration, calling a callable value and
     # normalizing the result to a String. Nil when no processor is configured.
     def configured_processor

data/patient_llm.gemspec

@@ -35,8 +35,8 @@ Gem::Specification.new do |spec|
 
   spec.require_paths = ["lib"]
 
-  spec.required_ruby_version = ">= 3.2"
+  spec.required_ruby_version = ">= 3.3"
 
-  spec.add_dependency "patient_http", ">= 1.5.0"
-  spec.add_dependency "prompt_builder", "~> 0.4"
+  spec.add_dependency "patient_http", ">= 1.8.0"
+  spec.add_dependency "prompt_builder", "~> 0.5"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: patient_llm
 version: !ruby/object:Gem::Version
-  version: 0.8.0
+  version: 0.10.0
 platform: ruby
 authors:
 - Brian Durand
@@ -15,28 +15,28 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.5.0
+        version: 1.8.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.5.0
+        version: 1.8.0
 - !ruby/object:Gem::Dependency
   name: prompt_builder
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: '0.5'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: '0.5'
 email:
 - bbdurand@gmail.com
 executables: []
@@ -75,7 +75,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '3.2'
+      version: '3.3'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="