patient_llm 0.7.0 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 74014452123a066e2e43338aea8a78473d4d1e76740ae9ef4067093c3af96c5c
4
- data.tar.gz: 1ae189616ab9daee96f8b8782da08871eb572024d4292c45c456d53de6706524
3
+ metadata.gz: 84d77f173d81a34af7c79a2a4052229916bf814660ef4a7616427601149bd8a9
4
+ data.tar.gz: 27f29411ac6e24fbb4c27da41f2681c0cd80576d5c6cda3ccf133761cf8ef92a
5
5
  SHA512:
6
- metadata.gz: cb5627c92970f28ce39f1e3d7da045eeb5641270a1f0b9d22413c71e68c57bd987e2d3d56cdcca2b3e80ad3c0571ecdd22729d04d39a8bcba8442b90f546055b
7
- data.tar.gz: 56fb911160aa80525325c94d15a74a833610428bc67d7702c16737b8ec6c1671afa6a3d59fcb2c92b2a58a0daa0405be28981017cfca99106d5873233c53f93e
6
+ metadata.gz: a02bd347cfb0ff748157d25615a4a588b34ebd2c465ae62abef11dc50c2e3e347fa9c53986e2e1c549fa06353b6c9407a0faa978541bca17233ce3334d58f0d5
7
+ data.tar.gz: 8a7aa8f715eed1566716bc829b4a518d72c3dbc6e639d4564f9de40becbfceff36fbbf432b19ea2fea1a352a9f0304b6ca0e04f5e4fe1817e4f6aa1679a80d36
data/CHANGELOG.md CHANGED
@@ -4,6 +4,45 @@ All notable changes to this project will be documented in this file.
4
4
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
5
5
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
6
 
7
+ ## 0.10.0
8
+
9
+ ### Added
10
+
11
+ - Streaming responses. Set `stream: true` on a provider, pass `stream:` to `ask` (on `PatientLLM` or an agent, including `continue` and `ask!`), or declare `stream true` on an agent. The provider streams the response so data keeps flowing during long generations, and the streamed body is reassembled into a single `PromptBuilder::Response` before the callback runs, so callbacks, hooks, and tool loops are unchanged. Every API format is supported; Gemini and Bedrock Converse use their streaming endpoints by default when streaming.
12
+ - `read_timeout` provider and request option: the maximum number of seconds to wait for data on a streamed request. It is not applied to requests that are not streamed.
13
+ - `max_response_size` provider and request option to raise the PatientHttp response size limit for LLM requests, since streamed bodies are larger than JSON bodies.
14
+ - A stream that reports an error event, ends before its final event, or is malformed is sent to `on_error` (and an agent's `failed` hook) as a `PatientHttp::RequestError` with the error type `:stream_error`.
15
+
16
+ ### Changed
17
+
18
+ - `stream:` passed to an agent's `ask` or `preview_request` is a request option instead of a session option, so it is accepted by `continue` and is not saved in the conversation state.
19
+ - patient_http minimum version is now 1.8.0, which adds the `read_timeout` and per-request `max_response_size` options.
20
+ - Minimum Ruby version is now 3.3.
21
+
22
+ ## 0.9.0
23
+
24
+ ### Added
25
+
26
+ - `Agent.reasoning` accepts a block or callable argument like the other scalar declarations (e.g. `reasoning { LLMExperiment.reasoning_effort }`). It is evaluated each time a request is built, and its result is normalized with the same rules as a static value.
27
+ - `reasoning false` turns reasoning off with `session.think(false)` (serialized as e.g. `thinking: {type: "disabled"}` on Anthropic Messages and Anthropic models on Converse), as opposed to `reasoning nil`, which sends no reasoning configuration and leaves the API's default in place. An effort level can accompany it (`reasoning false, effort: :low`). `reasoning true` turns reasoning on with the API's default depth.
28
+ - `Agent.reasoning` also accepts a Hash of options (`{effort:}`, `{budget_tokens:}`, `{enabled: false}`), the form the reader returns.
29
+
30
+ ### Changed
31
+
32
+ - The per-request `reasoning:` option to `ask`, `ask!`, and `preview_request` is normalized the same way as the `reasoning` declaration, so `reasoning: :low`, `reasoning: "low"`, `reasoning: {effort: :low}`, `reasoning: {budget_tokens: 2048}`, and `reasoning: false` produce the same session reasoning as the equivalent declaration. Previously the value was set on the session as-is, so shorthands such as `reasoning: :low` set the session reasoning to a String instead of a configuration Hash. `reasoning: nil` still sends no reasoning configuration for the request, and a Hash with other keys is still used as the raw reasoning configuration.
33
+ - `Agent.reasoning` raises an `ArgumentError` for values that are not a reasoning configuration (such as a number) instead of converting them to an effort level string.
34
+ - prompt_builder minimum version is now 0.5, which adds the explicit reasoning off configuration used by `reasoning false`.
35
+
36
+ ## 0.8.0
37
+
38
+ ### Added
39
+
40
+ - Requests can name the PatientHttp processor that executes them. Pass `processor:` to `PatientLLM.ask` (or through agent `ask` options), or set a default for all LLM requests with `config.processor` (a Symbol, String, or callable). When neither is set, requests use the default processor. The processor is preserved across automatic tool-loop iterations. Requires patient_http 1.5.0.
41
+
42
+ ### Changed
43
+
44
+ - patient_http minimum version is now 1.5.0.
45
+
7
46
  ## 0.7.0
8
47
 
9
48
  ### Changed
data/README.md CHANGED
@@ -149,7 +149,7 @@ end
149
149
  PatientHttp.register_secret("custom.api_key") { ENV["CUSTOM_API_KEY"] }
150
150
  ```
151
151
 
152
- Provider options: `url`, `headers`, `serializer`, `path`, `params` (merged into every payload), `preprocessors`, `timeout`, `max_tool_iterations`, `api_key` (see [provider presets](#provider-presets)), and `region` (used by region-templated presets such as `:bedrock_runtime`).
152
+ Provider options: `url`, `headers`, `serializer`, `path`, `params` (merged into every payload), `preprocessors`, `timeout`, `stream`, `read_timeout`, `max_response_size` (see [streaming responses](#streaming-responses)), `max_tool_iterations`, `api_key` (see [provider presets](#provider-presets)), and `region` (used by region-templated presets such as `:bedrock_runtime`).
153
153
 
154
154
  Every provider option (except `name`, `preset`, and `api_key`) may also be a callable, evaluated each time the provider is looked up — i.e. on every request — so values can be generated dynamically at runtime (`api_key` is consumed at registration time to build the authentication header; a callable key is resolved on the processor side at dispatch):
155
155
 
@@ -163,6 +163,43 @@ end
163
163
 
164
164
  Validation of a callable's result (serializer names, authentication headers) happens at lookup time instead of registration time.
165
165
 
166
+ ### Naming the request processor
167
+
168
+ Job-system integrations for patient_http can run multiple named processors with different configurations — for example, one tuned for long-running LLM requests and another for quick webhook deliveries. Set `processor` to route every LLM request to a named processor; it can also be a callable, resolved on every request. When unset, requests use the default processor. A per-request `processor:` option on `ask` overrides the configured value.
169
+
170
+ ```ruby
171
+ PatientLLM.configure do |config|
172
+ config.processor = :llm
173
+ end
174
+ ```
175
+
176
+ ### Streaming responses
177
+
178
+ LLM requests that generate for a long time can fail without streaming: the connection sits idle while the model works, and proxies, NAT gateways, or the provider's own front end may drop it before the response arrives. Anthropic, for example, recommends streaming for any request that might take longer than 10 minutes. With `stream: true`, the provider sends the response as it is generated, so data keeps flowing for the whole request.
179
+
180
+ The streamed body is reassembled into the same response the non-streaming endpoint would have returned before your callback runs (with prompt_builder's `Response.parse_stream`), so callbacks, hooks, tool loops, and structured output work the same way. Tokens are not delivered to your code as they arrive.
181
+
182
+ ```ruby
183
+ PatientLLM.configure do |config|
184
+ config.provider :anthropic, preset: :anthropic, api_key: -> { ENV["ANTHROPIC_API_KEY"] },
185
+ stream: true,
186
+ timeout: 30 * 60, # limit for the full stream
187
+ read_timeout: 120, # limit for each wait for data
188
+ max_response_size: 16 * 1024 * 1024 # streamed bodies are larger than JSON bodies
189
+ end
190
+ ```
191
+
192
+ - `stream` turns streaming on for the provider's requests. A `stream:` option on `ask` (or an agent's `stream` declaration) overrides it for a request, including `stream: false`. A `stream` key in `params` is treated the same way: request settings win over provider settings, and at each level the `stream` option wins over a `stream` param. When none is set, the session's `stream` field applies.
193
+ - `timeout` still limits the full request, so raise it for long generations.
194
+ - `read_timeout` is the maximum number of seconds to wait for data from the server. The timer starts again each time a chunk arrives, so a stream that stops fails fast while a long stream that keeps sending data runs until `timeout`. It only applies to streamed requests: without streaming, the server sends nothing until it has generated the whole response.
195
+ - `max_response_size` overrides the PatientHttp `max_response_size` (1MB by default) for the provider's requests. A streamed body is several times larger than the equivalent JSON body.
196
+
197
+ Streaming works with every API format. The Chat Completions, Open Responses, and Messages formats send `stream: true` in the request body. Gemini and Bedrock Converse stream from a different endpoint, which is used by default when streaming (`v1beta/models/{model}:streamGenerateContent?alt=sse` and `model/{model}/converse-stream`); an explicit `path` is used as is.
198
+
199
+ A stream that reports an error after the response started (such as an Anthropic `overloaded_error` event, which arrives with a 200 status), ends before its final event, or is malformed is sent to `on_error` (or an agent's `failed` hook) as a `PatientHttp::RequestError` with an `error_type` of `:stream_error`. The `error_class` names the prompt_builder error (`PromptBuilder::ErrorResponseError`, `PromptBuilder::IncompleteStreamError`, or `PromptBuilder::ParseError`), and `http_response` holds the streamed body.
200
+
201
+ Streaming requires patient_http 1.8.0 or later.
202
+
166
203
  ### Request signing (preprocessors)
167
204
 
168
205
  Some providers require request signing rather than a static authentication header — for example, AWS Bedrock with SigV4, where a signature is computed over the final outgoing request. For these, register a [request preprocessor](https://github.com/bdurand/patient_http#request-preprocessors) on the PatientHttp configuration and reference it by name from the provider. Like secrets, only the preprocessor name is serialized into the job queue; the signing logic and credentials stay on the processor side.
@@ -203,17 +240,31 @@ class ResearchAgent < PatientLLM::Agent
203
240
  max_output_tokens 4_000
204
241
  reasoning :medium # portable effort level, or reasoning budget_tokens: 8_000
205
242
  max_tool_iterations 5 # tool loop cap for this agent (default 10)
243
+ stream true # stream responses (see "Streaming responses")
206
244
  extra guardrail_config: {guardrailIdentifier: "gr-1", guardrailVersion: "1"}
207
245
  end
208
246
  ```
209
247
 
210
248
  `instructions` sets system-level instructions for the request. On APIs without a separate instructions field they are appended to the system prompt.
211
249
 
250
+ `reasoning` configures reasoning (extended thinking), applied with prompt_builder's `session.think`. It accepts a portable effort level (`:minimal`, `:low`, `:medium`, `:high`, `:xhigh`, `:max`), `effort:` or `budget_tokens:`, or `false` to turn reasoning off. `nil` and `false` are different: `reasoning nil` means no reasoning configuration, so the request carries no reasoning parameter and the API's default applies (some models, such as recent Claude models, reason by default), while `reasoning false` explicitly turns reasoning off (e.g. `thinking: {type: "disabled"}` for Claude models). An effort level can accompany `false` (`reasoning false, effort: :low`) for APIs such as Anthropic Messages, where effort also limits token spend without thinking; APIs whose only reasoning control is effort ignore it and keep reasoning off.
251
+
252
+ ```ruby
253
+ reasoning :medium # portable effort level
254
+ reasoning budget_tokens: 8_000 # explicit thinking budget, where supported
255
+ reasoning true # reasoning on, depth left to the API
256
+ reasoning false # reasoning off
257
+ reasoning false, effort: :low # reasoning off, with an effort level on Messages
258
+ reasoning nil # no reasoning configuration (removes an inherited value)
259
+ ```
260
+
261
+ The reader returns the normalized `session.think` options (`{effort: "medium"}`, `{budget_tokens: 8000}`, `{}`, `{enabled: false}`, or `nil`).
262
+
212
263
  `extra` sets provider-specific extra data on every session the agent builds. The recognized keys depend on the serializer used for the request — for example, the Bedrock Converse serializer maps `guardrail_config`, `stop_sequences`, `additional_model_request_fields`, and `performance_config` into the request payload (see the prompt_builder docs for each serializer's keys). Redeclaring `extra` in a subclass replaces the whole hash, and `continue` re-applies the agent's current value to restored sessions. A per-request `ask(extra: {...})` replaces the declaration for that request; to add to it instead, merge explicitly: `MyAgent.ask(msg, extra: MyAgent.extra.merge(...))`. Session options such as `extra:` are only accepted when `ask` builds the session — passing them to `continue`, or together with a `session:`, raises `ArgumentError`.
213
264
 
214
265
  ### Dynamic values
215
266
 
216
- Every scalar declaration (`provider`, `model`, `system`, `instructions`, `temperature`, `max_output_tokens`, `max_tool_iterations`, and `extra`) also accepts a block in lieu of the argument, or a callable (anything responding to `call`) as the argument. The block or callable is evaluated each time the agent builds or sends a request, so the value can be generated dynamically at runtime:
267
+ Every scalar declaration (`provider`, `model`, `system`, `instructions`, `temperature`, `max_output_tokens`, `reasoning`, `max_tool_iterations`, and `extra`) also accepts a block in lieu of the argument, or a callable (anything responding to `call`) as the argument. The block or callable is evaluated each time the agent builds or sends a request, so the value can be generated dynamically at runtime:
217
268
 
218
269
  ```ruby
219
270
  class ResearchAgent < PatientLLM::Agent
@@ -222,7 +273,21 @@ class ResearchAgent < PatientLLM::Agent
222
273
  end
223
274
  ```
224
275
 
225
- Validation and coercion (`extra`'s Hash check, `provider`'s symbol coercion) apply to the block's result each time it is evaluated.
276
+ Validation and coercion (`extra`'s Hash check, `provider`'s symbol coercion, `reasoning`'s normalization) apply to the block's result each time it is evaluated. A `reasoning` block can return any value the static form accepts — an effort level, a Hash such as `{budget_tokens: 2048}`, `false` to turn reasoning off, or `nil` for no configuration:
277
+
278
+ ```ruby
279
+ class SupportAgent < PatientLLM::Agent
280
+ model { LLMExperiment.model }
281
+ reasoning { LLMExperiment.reasoning_effort } # e.g. :low, :high, or false
282
+ max_output_tokens { LLMExperiment.max_output_tokens }
283
+ end
284
+ ```
285
+
286
+ Per-request session options override the declarations for one request. The `reasoning:` option is normalized the same way as the declaration, so `reasoning: :low`, `reasoning: {budget_tokens: 2048}`, and `reasoning: false` produce the same request as the equivalent `reasoning` declaration, and `reasoning: nil` sends no reasoning configuration for that request. Any other Hash (e.g. `{summary: "auto"}`) is set as the session's raw reasoning configuration for provider-specific settings:
287
+
288
+ ```ruby
289
+ SupportAgent.ask(message, reasoning: false, max_output_tokens: 1_000)
290
+ ```
226
291
 
227
292
  ### Inheritance
228
293
 
@@ -455,7 +520,11 @@ PatientLLM.ask(session,
455
520
  params: {max_completion_tokens: 1000}, # Additional request parameters
456
521
  preprocessors: :aws_sigv4, # Replace the provider's request preprocessors
457
522
  timeout: 600, # Request timeout in seconds
458
- max_tool_iterations: 3 # Tool loop cap for this request
523
+ stream: true, # Stream the response (see "Streaming responses")
524
+ read_timeout: 120, # Maximum wait for data on a streamed request
525
+ max_response_size: 16 * 1024 * 1024, # Maximum response body size in bytes
526
+ max_tool_iterations: 3, # Tool loop cap for this request
527
+ processor: :llm # Named PatientHttp processor for this request
459
528
  )
460
529
  ```
461
530
 
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.7.0
1
+ 0.10.0
@@ -68,6 +68,17 @@ module PatientLLM
68
68
  NOT_SPECIFIED = Object.new
69
69
  private_constant :NOT_SPECIFIED
70
70
 
71
+ # Keys accepted in a Hash reasoning configuration; they map to the
72
+ # arguments of `session.think`.
73
+ REASONING_OPTIONS = %i[enabled effort budget_tokens].freeze
74
+ private_constant :REASONING_OPTIONS
75
+
76
+ # Session options accepted by {ask} and {preview_request}. The session's
77
+ # stream field is excluded: stream: is a request option, so it can be
78
+ # used with {continue} and is not saved in the conversation state.
79
+ SESSION_OPTIONS = (PromptBuilder::Session::INITIALIZE_OPTIONS - [:stream]).freeze
80
+ private_constant :SESSION_OPTIONS
81
+
71
82
  class << self
72
83
  # DSL: get or set the provider name for this agent. The value is
73
84
  # inherited from the parent agent class unless set; pass an explicit nil
@@ -131,30 +142,48 @@ module PatientLLM
131
142
  get_or_set_setting(:temperature, value, block)
132
143
  end
133
144
 
134
- # DSL: get or set the reasoning configuration. Accepts a portable effort
135
- # level (:minimal, :low, :medium, :high, :xhigh, :max) or explicit options
136
- # (effort: or budget_tokens:) which are applied with `session.think`.
137
- # The value is inherited from the parent agent class unless set; pass an
138
- # explicit nil to remove an inherited value.
145
+ # DSL: get or set the reasoning configuration, applied to sessions with
146
+ # `session.think`. Accepts a portable effort level (:minimal, :low,
147
+ # :medium, :high, :xhigh, :max), explicit options (effort: or
148
+ # budget_tokens:), or false to turn reasoning off; an effort: may
149
+ # accompany false for APIs that limit token spend with it while
150
+ # thinking is off. True turns reasoning on and leaves the depth to the
151
+ # API. A Hash of the same options (:effort, :budget_tokens, :enabled) is
152
+ # also accepted, which is the form the reader returns.
153
+ #
154
+ # Nil and false are different: nil means no reasoning configuration,
155
+ # so the request carries no reasoning parameter and the API's default
156
+ # applies (some models reason by default), while false explicitly turns
157
+ # reasoning off. The value is inherited from the parent agent class
158
+ # unless set; pass an explicit nil to remove an inherited value.
159
+ #
160
+ # @example
161
+ # reasoning :medium
162
+ # reasoning budget_tokens: 8_000
163
+ # reasoning false
164
+ # reasoning { ReasoningSettings.effort } # e.g. :low, :high, or false
139
165
  #
140
- # @param value [Symbol, String, nil] a portable effort level.
166
+ # @param value [Symbol, String, Hash, Boolean, nil] a portable effort
167
+ # level, a Hash of options, or true/false to turn reasoning on or off.
141
168
  # Passing an explicit nil removes the previously set value.
142
169
  # @param effort [Symbol, String, nil] explicit effort level
143
170
  # @param budget_tokens [Integer, nil] explicit thinking token budget
144
- # @return [Hash, nil] the reasoning options
145
- def reasoning(value = NOT_SPECIFIED, effort: nil, budget_tokens: nil)
146
- return inherited_setting(:reasoning) if value.equal?(NOT_SPECIFIED) && effort.nil? && budget_tokens.nil?
147
-
148
- passed = [(value.equal?(NOT_SPECIFIED) ? nil : value), effort, budget_tokens].compact
149
- raise ArgumentError, "pass a level, effort:, or budget_tokens: — not more than one" if passed.size > 1
150
-
151
- @reasoning = if passed.empty?
152
- nil
153
- elsif effort || budget_tokens
154
- {effort: effort&.to_s, budget_tokens: budget_tokens}.compact
155
- else
156
- {effort: value.to_s}
171
+ # @yield an optional block (or callable argument) called each time the
172
+ # value is read so it can be generated dynamically; the result accepts
173
+ # the same forms as +value+ and is normalized at read time.
174
+ # @return [Hash, nil] the `session.think` options: +{effort:}+,
175
+ # +{budget_tokens:}+, +{}+ for API defaults, or +{enabled: false}+ (with an
176
+ # optional effort:) for reasoning off; nil for no configuration
177
+ # @raise [ArgumentError] if more than one of a level, effort:, or
178
+ # budget_tokens: is passed, or budget_tokens: is passed with false
179
+ def reasoning(value = NOT_SPECIFIED, effort: nil, budget_tokens: nil, &block)
180
+ if effort || budget_tokens
181
+ raise ArgumentError, "pass either an argument or a block, not both" if block
182
+
183
+ value = normalize_reasoning(value.equal?(NOT_SPECIFIED) ? nil : value, effort: effort, budget_tokens: budget_tokens)
157
184
  end
185
+
186
+ get_or_set_setting(:reasoning, value, block) { |resolved| normalize_reasoning(resolved) }
158
187
  end
159
188
 
160
189
  # DSL: get or set the maximum output tokens. The value is inherited from
@@ -181,6 +210,22 @@ module PatientLLM
181
210
  get_or_set_setting(:max_tool_iterations, value, block)
182
211
  end
183
212
 
213
+ # DSL: get or set whether this agent's requests stream their responses.
214
+ # The streamed body is reassembled into a single response before the
215
+ # hooks run, so streaming only changes how the response travels: data
216
+ # keeps flowing during long generations instead of the connection
217
+ # sitting idle. The value is inherited from the parent agent class
218
+ # unless set; pass an explicit nil to remove an inherited value and use
219
+ # the provider's setting. A stream: option passed to {ask} overrides it.
220
+ #
221
+ # @param value [Boolean, nil] whether to stream. Passing an explicit nil removes the previously set value.
222
+ # @yield an optional block (or callable argument) called each time the
223
+ # value is read so it can be generated dynamically.
224
+ # @return [Boolean, nil]
225
+ def stream(value = NOT_SPECIFIED, &block)
226
+ get_or_set_setting(:stream, value, block)
227
+ end
228
+
184
229
  # DSL: get or set provider-specific extra data set on sessions built by
185
230
  # this agent (e.g. the Bedrock Converse :guardrail_config). The
186
231
  # recognized keys depend on the serializer used for the request. The
@@ -276,19 +321,23 @@ module PatientLLM
276
321
  # agent. Defaults to sending the hooks to the agent itself.
277
322
  # @param options [Hash] per-request overrides forwarded to {PatientLLM.ask}
278
323
  # (url:, serializer:, path:, headers:, params:, preprocessors:, timeout:,
279
- # max_tool_iterations:), plus session options (model:, temperature:,
324
+ # stream:, read_timeout:, max_response_size:, max_tool_iterations:,
325
+ # processor:), plus session options (model:, temperature:,
280
326
  # extra:, etc. — any PromptBuilder::Session::INITIALIZE_OPTIONS key)
281
327
  # applied to the newly built session; session options raise
282
- # ArgumentError when a session: is supplied.
328
+ # ArgumentError when a session: is supplied. The reasoning: option
329
+ # accepts the same values as the {reasoning} DSL (e.g.
330
+ # +reasoning: :low+ or +reasoning: false+) and is normalized the same
331
+ # way; see {build_session}.
283
332
  # @return [Object] handler-specific identifier for the enqueued request
284
333
  def ask(message = nil, context: {}, session: nil, callback: nil, **options)
285
- session_options = options.slice(*PromptBuilder::Session::INITIALIZE_OPTIONS)
334
+ session_options = options.slice(*SESSION_OPTIONS)
286
335
  raise ArgumentError.new("session options cannot be passed when a session is provided") if session && session_options.any?
287
336
 
288
337
  session ||= build_session(**session_options)
289
338
  session.user(message) if message
290
339
 
291
- ask_options = options.except(*PromptBuilder::Session::INITIALIZE_OPTIONS)
340
+ ask_options = with_declared_stream(options.except(*SESSION_OPTIONS))
292
341
  iterations = max_tool_iterations
293
342
  ask_options[:max_tool_iterations] ||= iterations if iterations
294
343
 
@@ -371,7 +420,7 @@ module PatientLLM
371
420
  # @param options [Hash] the same per-request overrides accepted by {ask}
372
421
  # @return [PatientLLM::RequestPreview] the url, headers, and JSON payload
373
422
  def preview_request(message = nil, session: nil, context: nil, callback: nil, **options)
374
- session_options = options.slice(*PromptBuilder::Session::INITIALIZE_OPTIONS)
423
+ session_options = options.slice(*SESSION_OPTIONS)
375
424
  raise ArgumentError.new("session options cannot be passed when a session is provided") if session && session_options.any?
376
425
 
377
426
  session ||= build_session(**session_options)
@@ -380,7 +429,7 @@ module PatientLLM
380
429
  PatientLLM.preview_request(
381
430
  session,
382
431
  provider: provider_name!,
383
- **options.except(*PromptBuilder::Session::INITIALIZE_OPTIONS)
432
+ **with_declared_stream(options.except(*SESSION_OPTIONS))
384
433
  )
385
434
  end
386
435
 
@@ -388,10 +437,27 @@ module PatientLLM
388
437
  # the caller take precedence over the agent's declarations for the same
389
438
  # fields (pass an explicit nil to unset a declared value for one request).
390
439
  #
440
+ # The reasoning: option accepts the same values as the {reasoning} DSL
441
+ # and is normalized the same way, so +reasoning: :low+,
442
+ # +reasoning: {budget_tokens: 2048}+, and +reasoning: false+ produce the
443
+ # same session reasoning as the equivalent declarations. An explicit
444
+ # +reasoning: nil+ sends no reasoning configuration for the request. A
445
+ # Hash with keys other than :effort, :budget_tokens, and :enabled is
446
+ # used as the session's raw reasoning configuration for provider-specific
447
+ # settings.
448
+ #
449
+ # @param options [Hash] PromptBuilder::Session::INITIALIZE_OPTIONS values
391
450
  # @return [PromptBuilder::Session]
451
+ # @raise [ArgumentError] if the reasoning: option is invalid
392
452
  def build_session(**options)
453
+ overridden = options.keys
454
+ if options.key?(:reasoning) && !raw_reasoning_configuration?(options[:reasoning])
455
+ request_reasoning = normalize_reasoning(options.delete(:reasoning))
456
+ end
457
+
393
458
  session = PromptBuilder::Session.new(**options)
394
- apply_configuration(session, except: options.keys)
459
+ apply_reasoning(session, request_reasoning) if request_reasoning
460
+ apply_configuration(session, except: overridden)
395
461
  session
396
462
  end
397
463
 
@@ -410,13 +476,14 @@ module PatientLLM
410
476
  temperature_value = temperature
411
477
  max_output_tokens_value = max_output_tokens
412
478
  extra_value = extra
479
+ reasoning_value = reasoning
413
480
  apply_system_message(session, system_message) if system_message && !except.include?(:system)
414
481
  session.model = model_value if model_value && !except.include?(:model)
415
482
  session.instructions = instructions_value if instructions_value && !except.include?(:instructions)
416
483
  session.temperature = temperature_value if temperature_value && !except.include?(:temperature)
417
484
  session.max_output_tokens = max_output_tokens_value if max_output_tokens_value && !except.include?(:max_output_tokens)
418
485
  session.extra = extra_value if extra_value && !except.include?(:extra)
419
- session.think(**reasoning.transform_keys(&:to_sym)) if reasoning && !except.include?(:reasoning)
486
+ apply_reasoning(session, reasoning_value) if reasoning_value && !except.include?(:reasoning)
420
487
  if output_schema && !except.include?(:text)
421
488
  session.json_output(output_schema[:schema], name: output_schema[:name], strict: output_schema[:strict])
422
489
  end
@@ -445,6 +512,15 @@ module PatientLLM
445
512
  provider || raise(ArgumentError, "#{self} must declare a provider")
446
513
  end
447
514
 
515
+ # Add the agent's stream declaration to request options that do not set
516
+ # stream: themselves.
517
+ def with_declared_stream(options)
518
+ return options if options.key?(:stream)
519
+
520
+ declared = stream
521
+ declared.nil? ? options : options.merge(stream: declared)
522
+ end
523
+
448
524
  # Resolve the :callback option to the class name that travels through the
449
525
  # job queue. The class is resolved here so an unusable callback fails
450
526
  # where ask was called rather than later in a worker process.
@@ -485,6 +561,58 @@ module PatientLLM
485
561
  end
486
562
  end
487
563
 
564
+ # Normalize a reasoning value into the `session.think` options returned
565
+ # by {reasoning}: nil for no configuration, +{enabled: false}+ (with an
566
+ # optional effort) for reasoning off, and +{effort:}+, +{budget_tokens:}+,
567
+ # or +{}+ (API defaults) for reasoning on. Already-normalized values pass
568
+ # through unchanged.
569
+ def normalize_reasoning(value, effort: nil, budget_tokens: nil)
570
+ case value
571
+ when nil
572
+ return nil if effort.nil? && budget_tokens.nil?
573
+
574
+ normalize_reasoning(true, effort: effort, budget_tokens: budget_tokens)
575
+ when true
576
+ raise ArgumentError, "pass a level, effort:, or budget_tokens: — not more than one" if effort && budget_tokens
577
+
578
+ {effort: effort&.to_s, budget_tokens: budget_tokens}.compact
579
+ when false
580
+ raise ArgumentError, "budget_tokens: cannot be combined with reasoning false" if budget_tokens
581
+
582
+ {enabled: false, effort: effort&.to_s}.compact
583
+ when Symbol, String
584
+ raise ArgumentError, "pass a level, effort:, or budget_tokens: — not more than one" if effort || budget_tokens
585
+
586
+ {effort: value.to_s}
587
+ when Hash
588
+ raise ArgumentError, "pass reasoning options as a Hash or as keywords, not both" if effort || budget_tokens
589
+
590
+ options = value.transform_keys(&:to_sym)
591
+ unknown = options.keys - REASONING_OPTIONS
592
+ raise ArgumentError, "unknown reasoning options: #{unknown.join(", ")}" unless unknown.empty?
593
+
594
+ enabled = options.fetch(:enabled, true)
595
+ raise ArgumentError, "reasoning enabled: must be true or false" unless enabled == true || enabled == false
596
+
597
+ normalize_reasoning(enabled, effort: options[:effort], budget_tokens: options[:budget_tokens])
598
+ else
599
+ raise ArgumentError, "reasoning must be an effort level, a Hash of options, true, false, or nil; got #{value.inspect}"
600
+ end
601
+ end
602
+
603
+ # Whether a per-request reasoning value is a provider-specific Hash to
604
+ # set on the session as-is rather than options to normalize.
605
+ def raw_reasoning_configuration?(value)
606
+ value.is_a?(Hash) && value.keys.any? { |key| !REASONING_OPTIONS.include?(key.to_sym) }
607
+ end
608
+
609
+ # Apply normalized reasoning options (see {normalize_reasoning}) to a
610
+ # session with `session.think`.
611
+ def apply_reasoning(session, options)
612
+ options = options.dup
613
+ session.think(options.delete(:enabled) != false, **options)
614
+ end
615
+
488
616
  # Shared implementation of the scalar get-or-set DSL methods. With no
489
617
  # argument and no block, reads the setting; when the stored value
490
618
  # responds to #call it is called and the result used, so values can be
@@ -17,6 +17,12 @@ module PatientLLM
17
17
  # The user callback receives a `PromptBuilder::Response` object. Access
18
18
  # the response text via `response.text`, token usage via `response.usage`,
19
19
  # and model id via `response.model`.
20
+ #
21
+ # A streamed response body is reassembled into the same response the
22
+ # non-streaming endpoint would have returned. A stream that reports an error
23
+ # event, ends before its terminal event, or is malformed is routed to the
24
+ # user callback's on_error method as a `PatientHttp::RequestError` with the
25
+ # error type :stream_error.
20
26
  class Callback
21
27
  # Default maximum number of tool-execution rounds before the loop stops and
22
28
  # routes a `PatientHttp::RequestError` (error type :max_tool_iterations) to
@@ -87,9 +93,21 @@ module PatientLLM
87
93
  prepare_user_callback(user_callback, session: session, provider: provider_name, callback_args: callback_args, http_response: response, request_id: original_request_id)
88
94
 
89
95
  serializer = resolve_serializer(callback_args)
90
- # The headers let formats that return response metadata in HTTP headers
91
- # (e.g. the Converse response id) populate the parsed response.
92
- llm_response = PromptBuilder::Response.parse(response.json, serializer, headers: response.headers)
96
+ if streamed_response?(response, callback_args)
97
+ begin
98
+ # A response with no body (such as a 204) is an empty stream, which
99
+ # the parser reports as incomplete.
100
+ llm_response = PromptBuilder::Response.parse_stream(response.body || "", serializer, headers: response.headers)
101
+ rescue PromptBuilder::UnexpectedPayloadError, PromptBuilder::ParseError => e
102
+ error = request_error(e, :stream_error, response, original_request_id)
103
+ invoke_user_callback(user_callback, :on_error, session: session, provider: provider_name, callback_args: user_callback_args(callback_args), error: error, http_response: response, request_id: original_request_id)
104
+ return
105
+ end
106
+ else
107
+ # The headers let formats that return response metadata in HTTP headers
108
+ # (e.g. the Converse response id) populate the parsed response.
109
+ llm_response = PromptBuilder::Response.parse(response.json, serializer, headers: response.headers)
110
+ end
93
111
 
94
112
  if should_auto_execute_tools?(llm_response, user_callback)
95
113
  continue_tool_loop(session, provider_name, llm_response, callback_args, response, user_callback, original_request_id)
@@ -192,6 +210,18 @@ module PatientLLM
192
210
  provider_config&.dig(:serializer) || :chat_completion
193
211
  end
194
212
 
213
+ # Whether the response body is a stream. The stream flag recorded at
214
+ # dispatch covers requests streamed by request option, stream param,
215
+ # provider setting, or session field; the content type also catches an
216
+ # endpoint that streams without being asked, such as a custom path to a
217
+ # streaming endpoint.
218
+ def streamed_response?(response, callback_args)
219
+ return true if callback_args.fetch(:stream, nil) == true
220
+
221
+ content_type = response.headers["content-type"].to_s.downcase
222
+ STREAM_CONTENT_TYPES.any? { |type| content_type.start_with?(type) }
223
+ end
224
+
195
225
  def max_tool_iterations(callback_args)
196
226
  callback_args.fetch(:max_tool_iterations, MAX_TOOL_ITERATIONS).to_i
197
227
  end
@@ -274,11 +304,17 @@ module PatientLLM
274
304
  def max_tool_iterations_error(callback_args, http_response, request_id)
275
305
  limit = max_tool_iterations(callback_args)
276
306
  exception = MaxToolIterationsError.new("Tool-call loop exceeded #{limit} iterations")
307
+ request_error(exception, :max_tool_iterations, http_response, request_id)
308
+ end
309
+
310
+ # Wrap an exception raised while handling a response in a RequestError so
311
+ # it reaches the user callback's on_error method like a transport error.
312
+ def request_error(exception, error_type, http_response, request_id)
277
313
  PatientHttp::RequestError.new(
278
314
  class_name: exception.class.name,
279
315
  message: exception.message,
280
- backtrace: [],
281
- error_type: :max_tool_iterations,
316
+ backtrace: exception.backtrace || [],
317
+ error_type: error_type,
282
318
  duration: http_response&.duration,
283
319
  request_id: request_id,
284
320
  url: http_response&.url,
@@ -28,9 +28,18 @@ module PatientLLM
28
28
 
29
29
  attr_reader :session_offload_options
30
30
 
31
+ # Name of the PatientHttp processor that executes LLM requests, used when a
32
+ # request does not pass its own processor: option. The value can be a
33
+ # Symbol, a String, or a callable resolved on every request. When nil, the
34
+ # default processor is used.
35
+ #
36
+ # @return [Symbol, String, #call, nil]
37
+ attr_accessor :processor
38
+
31
39
  def initialize
32
40
  @providers = {}
33
41
  @session_offload_options = nil
42
+ @processor = nil
34
43
  end
35
44
 
36
45
  # Register a provider.
@@ -60,11 +69,18 @@ module PatientLLM
60
69
  # @param preprocessors [String, Symbol, Array<String, Symbol>, #call, nil] Names of request
61
70
  # preprocessors to apply to every request (e.g. for request signing). Names must be
62
71
  # registered on the PatientHttp configuration with `register_preprocessor`.
63
- # @param timeout [Numeric, #call, nil] Request timeout in seconds for this provider's requests
72
+ # @param timeout [Numeric, #call, nil] Request timeout in seconds for this provider's requests.
73
+ # For a streamed request this limits the full stream.
74
+ # @param stream [Boolean, #call, nil] Whether to stream this provider's responses. The
75
+ # streamed body is reassembled into a single response before the callback runs.
76
+ # @param read_timeout [Numeric, #call, nil] Maximum seconds to wait for data from the
77
+ # server on a streamed request. Not applied to requests that are not streamed.
78
+ # @param max_response_size [Integer, #call, nil] Maximum response body size in bytes
79
+ # for this provider's requests. A streamed body is larger than the equivalent JSON body.
64
80
  # @param max_tool_iterations [Integer, #call, nil] Maximum automatic tool-execution rounds
65
81
  # for this provider's requests (default: PatientLLM::Callback::MAX_TOOL_ITERATIONS)
66
82
  # @return [void]
67
- def provider(name, preset: nil, url: nil, api_key: nil, region: nil, headers: nil, serializer: nil, path: nil, params: {}, preprocessors: nil, timeout: nil, max_tool_iterations: nil)
83
+ def provider(name, preset: nil, url: nil, api_key: nil, region: nil, headers: nil, serializer: nil, path: nil, params: {}, preprocessors: nil, timeout: nil, stream: nil, read_timeout: nil, max_response_size: nil, max_tool_iterations: nil)
68
84
  preset_config = preset ? Presets.fetch(preset) : nil
69
85
  options = {
70
86
  preset_config: preset_config,
@@ -76,6 +92,9 @@ module PatientLLM
76
92
  params: params,
77
93
  preprocessors: preprocessors,
78
94
  timeout: timeout,
95
+ stream: stream,
96
+ read_timeout: read_timeout,
97
+ max_response_size: max_response_size,
79
98
  max_tool_iterations: max_tool_iterations
80
99
  }
81
100
 
@@ -158,6 +177,9 @@ module PatientLLM
158
177
  params: options[:params] || {},
159
178
  preprocessors: options[:preprocessors],
160
179
  timeout: options[:timeout],
180
+ stream: options[:stream],
181
+ read_timeout: options[:read_timeout],
182
+ max_response_size: options[:max_response_size],
161
183
  max_tool_iterations: options[:max_tool_iterations]
162
184
  }
163
185
  end
data/lib/patient_llm.rb CHANGED
@@ -31,6 +31,18 @@ module PatientLLM
31
31
  gemini: "v1beta/models/{model}:generateContent"
32
32
  }.freeze
33
33
 
34
+ # Default API paths per serializer format for streamed requests. Gemini and
35
+ # Bedrock Converse select streaming by endpoint; the other formats stream
36
+ # from the same endpoint with `stream: true` in the request body.
37
+ STREAMING_SERIALIZER_PATHS = SERIALIZER_PATHS.merge(
38
+ converse: "model/{model}/converse-stream",
39
+ gemini: "v1beta/models/{model}:streamGenerateContent?alt=sse"
40
+ ).freeze
41
+
42
+ # Content types of streamed response bodies: Server-Sent Events, and the AWS
43
+ # event-stream framing used by Bedrock Converse.
44
+ STREAM_CONTENT_TYPES = ["text/event-stream", "application/vnd.amazon.eventstream"].freeze
45
+
34
46
  # Required version header for the Anthropic Messages API.
35
47
  ANTHROPIC_VERSION = "2023-06-01"
36
48
 
@@ -153,11 +165,23 @@ module PatientLLM
153
165
  # preprocessors; pass an empty array to clear them for this request.
154
166
  # @param timeout [Numeric, nil] Request timeout in seconds for this request. Overrides
155
167
  # the provider's timeout; defaults to the PatientHttp processor configuration.
168
+ # For a streamed request this limits the full stream.
169
+ # @param stream [Boolean, nil] Whether to stream the response. The streamed body is
170
+ # reassembled into a single response before the callback runs. Overrides the
171
+ # provider's setting; when neither is set, the session's `stream` field applies.
172
+ # @param read_timeout [Numeric, nil] Maximum seconds to wait for data from the
173
+ # server on a streamed request. Overrides the provider's read timeout. Not
174
+ # applied to requests that are not streamed.
175
+ # @param max_response_size [Integer, nil] Maximum response body size in bytes.
176
+ # Overrides the provider's setting; defaults to the PatientHttp configuration.
156
177
  # @param max_tool_iterations [Integer, nil] Maximum automatic tool-execution rounds
157
178
  # for this request. Overrides the provider's setting; defaults to
158
179
  # {Callback::MAX_TOOL_ITERATIONS}.
180
+ # @param processor [String, Symbol, nil] Name of the PatientHttp processor that
181
+ # should execute this request. Overrides the configuration's processor;
182
+ # defaults to nil, which uses the default processor.
159
183
  # @return [Object] Handler-specific identifier for the enqueued request
160
- def ask(session, provider:, callback:, callback_args: {}, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, max_tool_iterations: nil)
184
+ def ask(session, provider:, callback:, callback_args: {}, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, stream: nil, read_timeout: nil, max_response_size: nil, max_tool_iterations: nil, processor: nil)
161
185
  request_options = build_request_options(
162
186
  url: url,
163
187
  serializer: serializer,
@@ -166,7 +190,11 @@ module PatientLLM
166
190
  params: params,
167
191
  preprocessors: preprocessors,
168
192
  timeout: timeout,
169
- max_tool_iterations: max_tool_iterations
193
+ stream: stream,
194
+ read_timeout: read_timeout,
195
+ max_response_size: max_response_size,
196
+ max_tool_iterations: max_tool_iterations,
197
+ processor: processor
170
198
  )
171
199
 
172
200
  dispatch(session, provider: provider, callback: callback, callback_args: callback_args, request_options: request_options)
@@ -193,10 +221,18 @@ module PatientLLM
193
221
  # @param preprocessors [String, Symbol, Array<String, Symbol>, nil] Accepted for
194
222
  # parity with {.ask}; preprocessors are not applied to the preview.
195
223
  # @param timeout [Numeric, nil] Accepted for parity with {.ask}; does not affect the preview.
224
+ # @param stream [Boolean, nil] Whether to stream the response. Streaming changes the
225
+ # endpoint path for some formats and the payload, so it is reflected in the preview.
226
+ # @param read_timeout [Numeric, nil] Accepted for parity with {.ask}; does not affect
227
+ # the preview.
228
+ # @param max_response_size [Integer, nil] Accepted for parity with {.ask}; does not
229
+ # affect the preview.
196
230
  # @param max_tool_iterations [Integer, nil] Accepted for parity with {.ask}; does not
197
231
  # affect the preview.
232
+ # @param processor [String, Symbol, nil] Accepted for parity with {.ask}; does not
233
+ # affect the preview.
198
234
  # @return [RequestPreview] The url, headers, and JSON payload the request would send
199
- def preview_request(session, provider:, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, max_tool_iterations: nil)
235
+ def preview_request(session, provider:, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, stream: nil, read_timeout: nil, max_response_size: nil, max_tool_iterations: nil, processor: nil)
200
236
  request_options = build_request_options(
201
237
  url: url,
202
238
  serializer: serializer,
@@ -205,7 +241,11 @@ module PatientLLM
205
241
  params: params,
206
242
  preprocessors: preprocessors,
207
243
  timeout: timeout,
208
- max_tool_iterations: max_tool_iterations
244
+ stream: stream,
245
+ read_timeout: read_timeout,
246
+ max_response_size: max_response_size,
247
+ max_tool_iterations: max_tool_iterations,
248
+ processor: processor
209
249
  )
210
250
 
211
251
  resolved = resolve_request(session, self.provider(provider) || {}, request_options)
@@ -238,6 +278,7 @@ module PatientLLM
238
278
  callback: callback.to_s,
239
279
  custom: PromptBuilder.jsonify(callback_args || {}),
240
280
  request_options: request_options,
281
+ stream: resolved.stream,
241
282
  max_tool_iterations: resolved.max_tool_iterations,
242
283
  tool_iteration: tool_iteration,
243
284
  original_request_id: original_request_id
@@ -250,7 +291,10 @@ module PatientLLM
250
291
  json: resolved.payload,
251
292
  headers: resolved.headers,
252
293
  preprocessors: resolved.preprocessors,
253
- timeout: resolved.timeout
294
+ timeout: resolved.timeout,
295
+ read_timeout: resolved.read_timeout,
296
+ max_response_size: resolved.max_response_size,
297
+ processor: resolved.processor
254
298
  )
255
299
  PatientHttp.execute_inline(
256
300
  request: request,
@@ -265,6 +309,9 @@ module PatientLLM
265
309
  headers: resolved.headers,
266
310
  preprocessors: resolved.preprocessors,
267
311
  timeout: resolved.timeout,
312
+ read_timeout: resolved.read_timeout,
313
+ max_response_size: resolved.max_response_size,
314
+ processor: resolved.processor,
268
315
  raise_error_responses: true,
269
316
  callback: PatientLLM::Callback,
270
317
  callback_args: dispatch_callback_args
@@ -276,14 +323,14 @@ module PatientLLM
276
323
 
277
324
  # Fully resolved request produced by merging per-request options over the
278
325
  # provider configuration.
279
- ResolvedRequest = Data.define(:url, :serializer, :headers, :payload, :preprocessors, :timeout, :max_tool_iterations)
326
+ ResolvedRequest = Data.define(:url, :serializer, :headers, :payload, :preprocessors, :timeout, :stream, :read_timeout, :max_response_size, :max_tool_iterations, :processor)
280
327
  private_constant :ResolvedRequest
281
328
 
282
329
  # Normalize per-request overrides into a request options hash. The request
283
330
  # options travel through the job queue in the callback args, which only
284
331
  # permit JSON-native values; convert Symbols (e.g. serializer names,
285
332
  # preprocessor names, header/param values) to Strings up front.
286
- def build_request_options(url:, serializer:, path:, headers:, params:, preprocessors:, timeout:, max_tool_iterations:)
333
+ def build_request_options(url:, serializer:, path:, headers:, params:, preprocessors:, timeout:, stream:, read_timeout:, max_response_size:, max_tool_iterations:, processor:)
287
334
  request_options = {}
288
335
  request_options["url"] = url if url
289
336
  request_options["serializer"] = serializer.to_s if serializer
@@ -292,7 +339,12 @@ module PatientLLM
292
339
  request_options["params"] = params if params && !params.empty?
293
340
  request_options["preprocessors"] = preprocessors if preprocessors
294
341
  request_options["timeout"] = timeout if timeout
342
+ # An explicit false is kept so a request can turn off a provider's streaming.
343
+ request_options["stream"] = stream unless stream.nil?
344
+ request_options["read_timeout"] = read_timeout if read_timeout
345
+ request_options["max_response_size"] = max_response_size if max_response_size
295
346
  request_options["max_tool_iterations"] = max_tool_iterations if max_tool_iterations
347
+ request_options["processor"] = processor.to_s if processor
296
348
 
297
349
  PromptBuilder.jsonify(request_options)
298
350
  end
@@ -308,7 +360,11 @@ module PatientLLM
308
360
  resolved_serializer = (request_options["serializer"] || provider_config[:serializer] || :chat_completion).to_sym
309
361
  validate_serializer!(resolved_serializer)
310
362
 
311
- resolved_path = request_options["path"] || provider_config[:path] || SERIALIZER_PATHS[resolved_serializer]
363
+ provider_params = deep_stringify_keys(provider_config[:params])
364
+ request_params = deep_stringify_keys(request_options["params"])
365
+ stream = resolve_stream(session, provider_config, request_options, provider_params, request_params)
366
+ default_paths = stream ? STREAMING_SERIALIZER_PATHS : SERIALIZER_PATHS
367
+ resolved_path = request_options["path"] || provider_config[:path] || default_paths[resolved_serializer]
312
368
  if resolved_path.include?("{model}")
313
369
  raise ArgumentError, "The endpoint path #{resolved_path.inspect} includes a {model} placeholder but session.model is not set" if session.model.nil?
314
370
 
@@ -323,10 +379,13 @@ module PatientLLM
323
379
  resolved_headers = {"anthropic-version" => ANTHROPIC_VERSION}.merge(resolved_headers)
324
380
  end
325
381
 
326
- resolved_params = (provider_config[:params] || {}).merge(request_options["params"] || {})
382
+ # The stream param is resolved with the stream option above and set
383
+ # through the serializer, which adds the options streaming needs (such
384
+ # as the Chat Completions usage option), so it is not merged on top.
385
+ resolved_params = provider_params.merge(request_params).except("stream")
327
386
 
328
- payload = session.request_payload(resolved_serializer)
329
- payload = deep_merge(payload, deep_stringify_keys(resolved_params)) unless resolved_params.empty?
387
+ payload = streaming_payload(session, resolved_serializer, stream)
388
+ payload = deep_merge(payload, resolved_params) unless resolved_params.empty?
330
389
 
331
390
  ResolvedRequest.new(
332
391
  url: join_url(resolved_url, resolved_path),
@@ -335,10 +394,53 @@ module PatientLLM
335
394
  payload: payload,
336
395
  preprocessors: request_options["preprocessors"] || provider_config[:preprocessors],
337
396
  timeout: request_options["timeout"] || provider_config[:timeout],
338
- max_tool_iterations: (request_options["max_tool_iterations"] || provider_config[:max_tool_iterations] || Callback::MAX_TOOL_ITERATIONS).to_i
397
+ stream: stream,
398
+ # The read timeout covers the wait for the response headers, which a
399
+ # non-streamed LLM request spends generating the whole response, so it
400
+ # only applies to streamed requests.
401
+ read_timeout: stream ? (request_options["read_timeout"] || provider_config[:read_timeout]) : nil,
402
+ max_response_size: request_options["max_response_size"] || provider_config[:max_response_size],
403
+ max_tool_iterations: (request_options["max_tool_iterations"] || provider_config[:max_tool_iterations] || Callback::MAX_TOOL_ITERATIONS).to_i,
404
+ processor: request_options["processor"] || configured_processor
339
405
  )
340
406
  end
341
407
 
408
+ # Whether the request streams its response. The first value that is set
409
+ # wins: the request option, a stream param on the request, the provider
410
+ # setting, a stream param on the provider, then the session's stream field.
411
+ def resolve_stream(session, provider_config, request_options, provider_params, request_params)
412
+ values = [
413
+ request_options["stream"],
414
+ request_params["stream"],
415
+ provider_config[:stream],
416
+ provider_params["stream"],
417
+ session.stream
418
+ ]
419
+ values.find { |value| !value.nil? } == true
420
+ end
421
+
422
+ # Serialize the session with its stream field set to match the resolved
423
+ # streaming setting. The field is set on a shallow copy, so the setting
424
+ # does not become part of the conversation state and concurrent requests
425
+ # sharing the session do not see each other's setting. A session that never
426
+ # set the field keeps it unset when the request is not streamed.
427
+ def streaming_payload(session, serializer, stream)
428
+ value = stream || (session.stream.nil? ? nil : false)
429
+ return session.request_payload(serializer) if value == session.stream
430
+
431
+ copy = session.dup
432
+ copy.stream = value
433
+ copy.request_payload(serializer)
434
+ end
435
+
436
+ # The processor name from the configuration, calling a callable value and
437
+ # normalizing the result to a String. Nil when no processor is configured.
438
+ def configured_processor
439
+ value = @configuration&.processor
440
+ value = value.call if value.respond_to?(:call)
441
+ value&.to_s
442
+ end
443
+
342
444
  # Replace secret reference header values with placeholders so a preview
343
445
  # never resolves or exposes secret values.
344
446
  def redact_secret_headers(headers)
data/patient_llm.gemspec CHANGED
@@ -35,8 +35,8 @@ Gem::Specification.new do |spec|
35
35
 
36
36
  spec.require_paths = ["lib"]
37
37
 
38
- spec.required_ruby_version = ">= 3.2"
38
+ spec.required_ruby_version = ">= 3.3"
39
39
 
40
- spec.add_dependency "patient_http", "~> 1.3"
41
- spec.add_dependency "prompt_builder", "~> 0.4"
40
+ spec.add_dependency "patient_http", ">= 1.8.0"
41
+ spec.add_dependency "prompt_builder", "~> 0.5"
42
42
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: patient_llm
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.7.0
4
+ version: 0.10.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Brian Durand
@@ -13,30 +13,30 @@ dependencies:
13
13
  name: patient_http
14
14
  requirement: !ruby/object:Gem::Requirement
15
15
  requirements:
16
- - - "~>"
16
+ - - ">="
17
17
  - !ruby/object:Gem::Version
18
- version: '1.3'
18
+ version: 1.8.0
19
19
  type: :runtime
20
20
  prerelease: false
21
21
  version_requirements: !ruby/object:Gem::Requirement
22
22
  requirements:
23
- - - "~>"
23
+ - - ">="
24
24
  - !ruby/object:Gem::Version
25
- version: '1.3'
25
+ version: 1.8.0
26
26
  - !ruby/object:Gem::Dependency
27
27
  name: prompt_builder
28
28
  requirement: !ruby/object:Gem::Requirement
29
29
  requirements:
30
30
  - - "~>"
31
31
  - !ruby/object:Gem::Version
32
- version: '0.4'
32
+ version: '0.5'
33
33
  type: :runtime
34
34
  prerelease: false
35
35
  version_requirements: !ruby/object:Gem::Requirement
36
36
  requirements:
37
37
  - - "~>"
38
38
  - !ruby/object:Gem::Version
39
- version: '0.4'
39
+ version: '0.5'
40
40
  email:
41
41
  - bbdurand@gmail.com
42
42
  executables: []
@@ -75,7 +75,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
75
75
  requirements:
76
76
  - - ">="
77
77
  - !ruby/object:Gem::Version
78
- version: '3.2'
78
+ version: '3.3'
79
79
  required_rubygems_version: !ruby/object:Gem::Requirement
80
80
  requirements:
81
81
  - - ">="