patient_llm 0.7.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +39 -0
- data/README.md +73 -4
- data/VERSION +1 -1
- data/lib/patient_llm/agent.rb +155 -27
- data/lib/patient_llm/callback.rb +41 -5
- data/lib/patient_llm/configuration.rb +24 -2
- data/lib/patient_llm.rb +114 -12
- data/patient_llm.gemspec +3 -3
- metadata +8 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 84d77f173d81a34af7c79a2a4052229916bf814660ef4a7616427601149bd8a9
|
|
4
|
+
data.tar.gz: 27f29411ac6e24fbb4c27da41f2681c0cd80576d5c6cda3ccf133761cf8ef92a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a02bd347cfb0ff748157d25615a4a588b34ebd2c465ae62abef11dc50c2e3e347fa9c53986e2e1c549fa06353b6c9407a0faa978541bca17233ce3334d58f0d5
|
|
7
|
+
data.tar.gz: 8a7aa8f715eed1566716bc829b4a518d72c3dbc6e639d4564f9de40becbfceff36fbbf432b19ea2fea1a352a9f0304b6ca0e04f5e4fe1817e4f6aa1679a80d36
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,45 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
5
5
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## 0.10.0
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- Streaming responses. Set `stream: true` on a provider, pass `stream:` to `ask` (on `PatientLLM` or an agent, including `continue` and `ask!`), or declare `stream true` on an agent. The provider streams the response so data keeps flowing during long generations, and the streamed body is reassembled into a single `PromptBuilder::Response` before the callback runs, so callbacks, hooks, and tool loops are unchanged. Every API format is supported; Gemini and Bedrock Converse use their streaming endpoints by default when streaming.
|
|
12
|
+
- `read_timeout` provider and request option: the maximum number of seconds to wait for data on a streamed request. It is not applied to requests that are not streamed.
|
|
13
|
+
- `max_response_size` provider and request option to raise the PatientHttp response size limit for LLM requests, since streamed bodies are larger than JSON bodies.
|
|
14
|
+
- A stream that reports an error event, ends before its final event, or is malformed is sent to `on_error` (and an agent's `failed` hook) as a `PatientHttp::RequestError` with the error type `:stream_error`.
|
|
15
|
+
|
|
16
|
+
### Changed
|
|
17
|
+
|
|
18
|
+
- `stream:` passed to an agent's `ask` or `preview_request` is a request option instead of a session option, so it is accepted by `continue` and is not saved in the conversation state.
|
|
19
|
+
- patient_http minimum version is now 1.8.0, which adds the `read_timeout` and per-request `max_response_size` options.
|
|
20
|
+
- Minimum Ruby version is now 3.3.
|
|
21
|
+
|
|
22
|
+
## 0.9.0
|
|
23
|
+
|
|
24
|
+
### Added
|
|
25
|
+
|
|
26
|
+
- `Agent.reasoning` accepts a block or callable argument like the other scalar declarations (e.g. `reasoning { LLMExperiment.reasoning_effort }`). It is evaluated each time a request is built, and its result is normalized with the same rules as a static value.
|
|
27
|
+
- `reasoning false` turns reasoning off with `session.think(false)` (serialized as e.g. `thinking: {type: "disabled"}` on Anthropic Messages and Anthropic models on Converse), as opposed to `reasoning nil`, which sends no reasoning configuration and leaves the API's default in place. An effort level can accompany it (`reasoning false, effort: :low`). `reasoning true` turns reasoning on with the API's default depth.
|
|
28
|
+
- `Agent.reasoning` also accepts a Hash of options (`{effort:}`, `{budget_tokens:}`, `{enabled: false}`), the form the reader returns.
|
|
29
|
+
|
|
30
|
+
### Changed
|
|
31
|
+
|
|
32
|
+
- The per-request `reasoning:` option to `ask`, `ask!`, and `preview_request` is normalized the same way as the `reasoning` declaration, so `reasoning: :low`, `reasoning: "low"`, `reasoning: {effort: :low}`, `reasoning: {budget_tokens: 2048}`, and `reasoning: false` produce the same session reasoning as the equivalent declaration. Previously the value was set on the session as-is, so shorthands such as `reasoning: :low` set the session reasoning to a String instead of a configuration Hash. `reasoning: nil` still sends no reasoning configuration for the request, and a Hash with other keys is still used as the raw reasoning configuration.
|
|
33
|
+
- `Agent.reasoning` raises an `ArgumentError` for values that are not a reasoning configuration (such as a number) instead of converting them to an effort level string.
|
|
34
|
+
- prompt_builder minimum version is now 0.5, which adds the explicit reasoning off configuration used by `reasoning false`.
|
|
35
|
+
|
|
36
|
+
## 0.8.0
|
|
37
|
+
|
|
38
|
+
### Added
|
|
39
|
+
|
|
40
|
+
- Requests can name the PatientHttp processor that executes them. Pass `processor:` to `PatientLLM.ask` (or through agent `ask` options), or set a default for all LLM requests with `config.processor` (a Symbol, String, or callable). When neither is set, requests use the default processor. The processor is preserved across automatic tool-loop iterations. Requires patient_http 1.5.0.
|
|
41
|
+
|
|
42
|
+
### Changed
|
|
43
|
+
|
|
44
|
+
- patient_http minimum version is now 1.5.0.
|
|
45
|
+
|
|
7
46
|
## 0.7.0
|
|
8
47
|
|
|
9
48
|
### Changed
|
data/README.md
CHANGED
|
@@ -149,7 +149,7 @@ end
|
|
|
149
149
|
PatientHttp.register_secret("custom.api_key") { ENV["CUSTOM_API_KEY"] }
|
|
150
150
|
```
|
|
151
151
|
|
|
152
|
-
Provider options: `url`, `headers`, `serializer`, `path`, `params` (merged into every payload), `preprocessors`, `timeout`, `max_tool_iterations`, `api_key` (see [provider presets](#provider-presets)), and `region` (used by region-templated presets such as `:bedrock_runtime`).
|
|
152
|
+
Provider options: `url`, `headers`, `serializer`, `path`, `params` (merged into every payload), `preprocessors`, `timeout`, `stream`, `read_timeout`, `max_response_size` (see [streaming responses](#streaming-responses)), `max_tool_iterations`, `api_key` (see [provider presets](#provider-presets)), and `region` (used by region-templated presets such as `:bedrock_runtime`).
|
|
153
153
|
|
|
154
154
|
Every provider option (except `name`, `preset`, and `api_key`) may also be a callable, evaluated each time the provider is looked up — i.e. on every request — so values can be generated dynamically at runtime (`api_key` is consumed at registration time to build the authentication header; a callable key is resolved on the processor side at dispatch):
|
|
155
155
|
|
|
@@ -163,6 +163,43 @@ end
|
|
|
163
163
|
|
|
164
164
|
Validation of a callable's result (serializer names, authentication headers) happens at lookup time instead of registration time.
|
|
165
165
|
|
|
166
|
+
### Naming the request processor
|
|
167
|
+
|
|
168
|
+
Job-system integrations for patient_http can run multiple named processors with different configurations — for example, one tuned for long-running LLM requests and another for quick webhook deliveries. Set `processor` to route every LLM request to a named processor; it can also be a callable, resolved on every request. When unset, requests use the default processor. A per-request `processor:` option on `ask` overrides the configured value.
|
|
169
|
+
|
|
170
|
+
```ruby
|
|
171
|
+
PatientLLM.configure do |config|
|
|
172
|
+
config.processor = :llm
|
|
173
|
+
end
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
### Streaming responses
|
|
177
|
+
|
|
178
|
+
LLM requests that generate for a long time can fail without streaming: the connection sits idle while the model works, and proxies, NAT gateways, or the provider's own front end may drop it before the response arrives. Anthropic, for example, recommends streaming for any request that might take longer than 10 minutes. With `stream: true`, the provider sends the response as it is generated, so data keeps flowing for the whole request.
|
|
179
|
+
|
|
180
|
+
The streamed body is reassembled into the same response the non-streaming endpoint would have returned before your callback runs (with prompt_builder's `Response.parse_stream`), so callbacks, hooks, tool loops, and structured output work the same way. Tokens are not delivered to your code as they arrive.
|
|
181
|
+
|
|
182
|
+
```ruby
|
|
183
|
+
PatientLLM.configure do |config|
|
|
184
|
+
config.provider :anthropic, preset: :anthropic, api_key: -> { ENV["ANTHROPIC_API_KEY"] },
|
|
185
|
+
stream: true,
|
|
186
|
+
timeout: 30 * 60, # limit for the full stream
|
|
187
|
+
read_timeout: 120, # limit for each wait for data
|
|
188
|
+
max_response_size: 16 * 1024 * 1024 # streamed bodies are larger than JSON bodies
|
|
189
|
+
end
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
- `stream` turns streaming on for the provider's requests. A `stream:` option on `ask` (or an agent's `stream` declaration) overrides it for a request, including `stream: false`. A `stream` key in `params` is treated the same way: request settings win over provider settings, and at each level the `stream` option wins over a `stream` param. When none is set, the session's `stream` field applies.
|
|
193
|
+
- `timeout` still limits the full request, so raise it for long generations.
|
|
194
|
+
- `read_timeout` is the maximum number of seconds to wait for data from the server. The timer starts again each time a chunk arrives, so a stream that stops fails fast while a long stream that keeps sending data runs until `timeout`. It only applies to streamed requests: without streaming, the server sends nothing until it has generated the whole response.
|
|
195
|
+
- `max_response_size` overrides the PatientHttp `max_response_size` (1MB by default) for the provider's requests. A streamed body is several times larger than the equivalent JSON body.
|
|
196
|
+
|
|
197
|
+
Streaming works with every API format. The Chat Completions, Open Responses, and Messages formats send `stream: true` in the request body. Gemini and Bedrock Converse stream from a different endpoint, which is used by default when streaming (`v1beta/models/{model}:streamGenerateContent?alt=sse` and `model/{model}/converse-stream`); an explicit `path` is used as is.
|
|
198
|
+
|
|
199
|
+
A stream that reports an error after the response started (such as an Anthropic `overloaded_error` event, which arrives with a 200 status), ends before its final event, or is malformed is sent to `on_error` (or an agent's `failed` hook) as a `PatientHttp::RequestError` with an `error_type` of `:stream_error`. The `error_class` names the prompt_builder error (`PromptBuilder::ErrorResponseError`, `PromptBuilder::IncompleteStreamError`, or `PromptBuilder::ParseError`), and `http_response` holds the streamed body.
|
|
200
|
+
|
|
201
|
+
Streaming requires patient_http 1.8.0 or later.
|
|
202
|
+
|
|
166
203
|
### Request signing (preprocessors)
|
|
167
204
|
|
|
168
205
|
Some providers require request signing rather than a static authentication header — for example, AWS Bedrock with SigV4, where a signature is computed over the final outgoing request. For these, register a [request preprocessor](https://github.com/bdurand/patient_http#request-preprocessors) on the PatientHttp configuration and reference it by name from the provider. Like secrets, only the preprocessor name is serialized into the job queue; the signing logic and credentials stay on the processor side.
|
|
@@ -203,17 +240,31 @@ class ResearchAgent < PatientLLM::Agent
|
|
|
203
240
|
max_output_tokens 4_000
|
|
204
241
|
reasoning :medium # portable effort level, or reasoning budget_tokens: 8_000
|
|
205
242
|
max_tool_iterations 5 # tool loop cap for this agent (default 10)
|
|
243
|
+
stream true # stream responses (see "Streaming responses")
|
|
206
244
|
extra guardrail_config: {guardrailIdentifier: "gr-1", guardrailVersion: "1"}
|
|
207
245
|
end
|
|
208
246
|
```
|
|
209
247
|
|
|
210
248
|
`instructions` sets system-level instructions for the request. On APIs without a separate instructions field they are appended to the system prompt.
|
|
211
249
|
|
|
250
|
+
`reasoning` configures reasoning (extended thinking), applied with prompt_builder's `session.think`. It accepts a portable effort level (`:minimal`, `:low`, `:medium`, `:high`, `:xhigh`, `:max`), `effort:` or `budget_tokens:`, or `false` to turn reasoning off. `nil` and `false` are different: `reasoning nil` means no reasoning configuration, so the request carries no reasoning parameter and the API's default applies (some models, such as recent Claude models, reason by default), while `reasoning false` explicitly turns reasoning off (e.g. `thinking: {type: "disabled"}` for Claude models). An effort level can accompany `false` (`reasoning false, effort: :low`) for APIs such as Anthropic Messages, where effort also limits token spend without thinking; APIs whose only reasoning control is effort ignore it and keep reasoning off.
|
|
251
|
+
|
|
252
|
+
```ruby
|
|
253
|
+
reasoning :medium # portable effort level
|
|
254
|
+
reasoning budget_tokens: 8_000 # explicit thinking budget, where supported
|
|
255
|
+
reasoning true # reasoning on, depth left to the API
|
|
256
|
+
reasoning false # reasoning off
|
|
257
|
+
reasoning false, effort: :low # reasoning off, with an effort level on Messages
|
|
258
|
+
reasoning nil # no reasoning configuration (removes an inherited value)
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
The reader returns the normalized `session.think` options (`{effort: "medium"}`, `{budget_tokens: 8000}`, `{}`, `{enabled: false}`, or `nil`).
|
|
262
|
+
|
|
212
263
|
`extra` sets provider-specific extra data on every session the agent builds. The recognized keys depend on the serializer used for the request — for example, the Bedrock Converse serializer maps `guardrail_config`, `stop_sequences`, `additional_model_request_fields`, and `performance_config` into the request payload (see the prompt_builder docs for each serializer's keys). Redeclaring `extra` in a subclass replaces the whole hash, and `continue` re-applies the agent's current value to restored sessions. A per-request `ask(extra: {...})` replaces the declaration for that request; to add to it instead, merge explicitly: `MyAgent.ask(msg, extra: MyAgent.extra.merge(...))`. Session options such as `extra:` are only accepted when `ask` builds the session — passing them to `continue`, or together with a `session:`, raises `ArgumentError`.
|
|
213
264
|
|
|
214
265
|
### Dynamic values
|
|
215
266
|
|
|
216
|
-
Every scalar declaration (`provider`, `model`, `system`, `instructions`, `temperature`, `max_output_tokens`, `max_tool_iterations`, and `extra`) also accepts a block in lieu of the argument, or a callable (anything responding to `call`) as the argument. The block or callable is evaluated each time the agent builds or sends a request, so the value can be generated dynamically at runtime:
|
|
267
|
+
Every scalar declaration (`provider`, `model`, `system`, `instructions`, `temperature`, `max_output_tokens`, `reasoning`, `max_tool_iterations`, and `extra`) also accepts a block in lieu of the argument, or a callable (anything responding to `call`) as the argument. The block or callable is evaluated each time the agent builds or sends a request, so the value can be generated dynamically at runtime:
|
|
217
268
|
|
|
218
269
|
```ruby
|
|
219
270
|
class ResearchAgent < PatientLLM::Agent
|
|
@@ -222,7 +273,21 @@ class ResearchAgent < PatientLLM::Agent
|
|
|
222
273
|
end
|
|
223
274
|
```
|
|
224
275
|
|
|
225
|
-
Validation and coercion (`extra`'s Hash check, `provider`'s symbol coercion) apply to the block's result each time it is evaluated.
|
|
276
|
+
Validation and coercion (`extra`'s Hash check, `provider`'s symbol coercion, `reasoning`'s normalization) apply to the block's result each time it is evaluated. A `reasoning` block can return any value the static form accepts — an effort level, a Hash such as `{budget_tokens: 2048}`, `false` to turn reasoning off, or `nil` for no configuration:
|
|
277
|
+
|
|
278
|
+
```ruby
|
|
279
|
+
class SupportAgent < PatientLLM::Agent
|
|
280
|
+
model { LLMExperiment.model }
|
|
281
|
+
reasoning { LLMExperiment.reasoning_effort } # e.g. :low, :high, or false
|
|
282
|
+
max_output_tokens { LLMExperiment.max_output_tokens }
|
|
283
|
+
end
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
Per-request session options override the declarations for one request. The `reasoning:` option is normalized the same way as the declaration, so `reasoning: :low`, `reasoning: {budget_tokens: 2048}`, and `reasoning: false` produce the same request as the equivalent `reasoning` declaration, and `reasoning: nil` sends no reasoning configuration for that request. Any other Hash (e.g. `{summary: "auto"}`) is set as the session's raw reasoning configuration for provider-specific settings:
|
|
287
|
+
|
|
288
|
+
```ruby
|
|
289
|
+
SupportAgent.ask(message, reasoning: false, max_output_tokens: 1_000)
|
|
290
|
+
```
|
|
226
291
|
|
|
227
292
|
### Inheritance
|
|
228
293
|
|
|
@@ -455,7 +520,11 @@ PatientLLM.ask(session,
|
|
|
455
520
|
params: {max_completion_tokens: 1000}, # Additional request parameters
|
|
456
521
|
preprocessors: :aws_sigv4, # Replace the provider's request preprocessors
|
|
457
522
|
timeout: 600, # Request timeout in seconds
|
|
458
|
-
|
|
523
|
+
stream: true, # Stream the response (see "Streaming responses")
|
|
524
|
+
read_timeout: 120, # Maximum wait for data on a streamed request
|
|
525
|
+
max_response_size: 16 * 1024 * 1024, # Maximum response body size in bytes
|
|
526
|
+
max_tool_iterations: 3, # Tool loop cap for this request
|
|
527
|
+
processor: :llm # Named PatientHttp processor for this request
|
|
459
528
|
)
|
|
460
529
|
```
|
|
461
530
|
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.
|
|
1
|
+
0.10.0
|
data/lib/patient_llm/agent.rb
CHANGED
|
@@ -68,6 +68,17 @@ module PatientLLM
|
|
|
68
68
|
NOT_SPECIFIED = Object.new
|
|
69
69
|
private_constant :NOT_SPECIFIED
|
|
70
70
|
|
|
71
|
+
# Keys accepted in a Hash reasoning configuration; they map to the
|
|
72
|
+
# arguments of `session.think`.
|
|
73
|
+
REASONING_OPTIONS = %i[enabled effort budget_tokens].freeze
|
|
74
|
+
private_constant :REASONING_OPTIONS
|
|
75
|
+
|
|
76
|
+
# Session options accepted by {ask} and {preview_request}. The session's
|
|
77
|
+
# stream field is excluded: stream: is a request option, so it can be
|
|
78
|
+
# used with {continue} and is not saved in the conversation state.
|
|
79
|
+
SESSION_OPTIONS = (PromptBuilder::Session::INITIALIZE_OPTIONS - [:stream]).freeze
|
|
80
|
+
private_constant :SESSION_OPTIONS
|
|
81
|
+
|
|
71
82
|
class << self
|
|
72
83
|
# DSL: get or set the provider name for this agent. The value is
|
|
73
84
|
# inherited from the parent agent class unless set; pass an explicit nil
|
|
@@ -131,30 +142,48 @@ module PatientLLM
|
|
|
131
142
|
get_or_set_setting(:temperature, value, block)
|
|
132
143
|
end
|
|
133
144
|
|
|
134
|
-
# DSL: get or set the reasoning configuration
|
|
135
|
-
# level (:minimal, :low,
|
|
136
|
-
#
|
|
137
|
-
#
|
|
138
|
-
#
|
|
145
|
+
# DSL: get or set the reasoning configuration, applied to sessions with
|
|
146
|
+
# `session.think`. Accepts a portable effort level (:minimal, :low,
|
|
147
|
+
# :medium, :high, :xhigh, :max), explicit options (effort: or
|
|
148
|
+
# budget_tokens:), or false to turn reasoning off; an effort: may
|
|
149
|
+
# accompany false for APIs that limit token spend with it while
|
|
150
|
+
# thinking is off. True turns reasoning on and leaves the depth to the
|
|
151
|
+
# API. A Hash of the same options (:effort, :budget_tokens, :enabled) is
|
|
152
|
+
# also accepted, which is the form the reader returns.
|
|
153
|
+
#
|
|
154
|
+
# Nil and false are different: nil means no reasoning configuration,
|
|
155
|
+
# so the request carries no reasoning parameter and the API's default
|
|
156
|
+
# applies (some models reason by default), while false explicitly turns
|
|
157
|
+
# reasoning off. The value is inherited from the parent agent class
|
|
158
|
+
# unless set; pass an explicit nil to remove an inherited value.
|
|
159
|
+
#
|
|
160
|
+
# @example
|
|
161
|
+
# reasoning :medium
|
|
162
|
+
# reasoning budget_tokens: 8_000
|
|
163
|
+
# reasoning false
|
|
164
|
+
# reasoning { ReasoningSettings.effort } # e.g. :low, :high, or false
|
|
139
165
|
#
|
|
140
|
-
# @param value [Symbol, String, nil] a portable effort
|
|
166
|
+
# @param value [Symbol, String, Hash, Boolean, nil] a portable effort
|
|
167
|
+
# level, a Hash of options, or true/false to turn reasoning on or off.
|
|
141
168
|
# Passing an explicit nil removes the previously set value.
|
|
142
169
|
# @param effort [Symbol, String, nil] explicit effort level
|
|
143
170
|
# @param budget_tokens [Integer, nil] explicit thinking token budget
|
|
144
|
-
# @
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
171
|
+
# @yield an optional block (or callable argument) called each time the
|
|
172
|
+
# value is read so it can be generated dynamically; the result accepts
|
|
173
|
+
# the same forms as +value+ and is normalized at read time.
|
|
174
|
+
# @return [Hash, nil] the `session.think` options: +{effort:}+,
|
|
175
|
+
# +{budget_tokens:}+, +{}+ for API defaults, or +{enabled: false}+ (with an
|
|
176
|
+
# optional effort:) for reasoning off; nil for no configuration
|
|
177
|
+
# @raise [ArgumentError] if more than one of a level, effort:, or
|
|
178
|
+
# budget_tokens: is passed, or budget_tokens: is passed with false
|
|
179
|
+
def reasoning(value = NOT_SPECIFIED, effort: nil, budget_tokens: nil, &block)
|
|
180
|
+
if effort || budget_tokens
|
|
181
|
+
raise ArgumentError, "pass either an argument or a block, not both" if block
|
|
182
|
+
|
|
183
|
+
value = normalize_reasoning(value.equal?(NOT_SPECIFIED) ? nil : value, effort: effort, budget_tokens: budget_tokens)
|
|
157
184
|
end
|
|
185
|
+
|
|
186
|
+
get_or_set_setting(:reasoning, value, block) { |resolved| normalize_reasoning(resolved) }
|
|
158
187
|
end
|
|
159
188
|
|
|
160
189
|
# DSL: get or set the maximum output tokens. The value is inherited from
|
|
@@ -181,6 +210,22 @@ module PatientLLM
|
|
|
181
210
|
get_or_set_setting(:max_tool_iterations, value, block)
|
|
182
211
|
end
|
|
183
212
|
|
|
213
|
+
# DSL: get or set whether this agent's requests stream their responses.
|
|
214
|
+
# The streamed body is reassembled into a single response before the
|
|
215
|
+
# hooks run, so streaming only changes how the response travels: data
|
|
216
|
+
# keeps flowing during long generations instead of the connection
|
|
217
|
+
# sitting idle. The value is inherited from the parent agent class
|
|
218
|
+
# unless set; pass an explicit nil to remove an inherited value and use
|
|
219
|
+
# the provider's setting. A stream: option passed to {ask} overrides it.
|
|
220
|
+
#
|
|
221
|
+
# @param value [Boolean, nil] whether to stream. Passing an explicit nil removes the previously set value.
|
|
222
|
+
# @yield an optional block (or callable argument) called each time the
|
|
223
|
+
# value is read so it can be generated dynamically.
|
|
224
|
+
# @return [Boolean, nil]
|
|
225
|
+
def stream(value = NOT_SPECIFIED, &block)
|
|
226
|
+
get_or_set_setting(:stream, value, block)
|
|
227
|
+
end
|
|
228
|
+
|
|
184
229
|
# DSL: get or set provider-specific extra data set on sessions built by
|
|
185
230
|
# this agent (e.g. the Bedrock Converse :guardrail_config). The
|
|
186
231
|
# recognized keys depend on the serializer used for the request. The
|
|
@@ -276,19 +321,23 @@ module PatientLLM
|
|
|
276
321
|
# agent. Defaults to sending the hooks to the agent itself.
|
|
277
322
|
# @param options [Hash] per-request overrides forwarded to {PatientLLM.ask}
|
|
278
323
|
# (url:, serializer:, path:, headers:, params:, preprocessors:, timeout:,
|
|
279
|
-
#
|
|
324
|
+
# stream:, read_timeout:, max_response_size:, max_tool_iterations:,
|
|
325
|
+
# processor:), plus session options (model:, temperature:,
|
|
280
326
|
# extra:, etc. — any PromptBuilder::Session::INITIALIZE_OPTIONS key)
|
|
281
327
|
# applied to the newly built session; session options raise
|
|
282
|
-
# ArgumentError when a session: is supplied.
|
|
328
|
+
# ArgumentError when a session: is supplied. The reasoning: option
|
|
329
|
+
# accepts the same values as the {reasoning} DSL (e.g.
|
|
330
|
+
# +reasoning: :low+ or +reasoning: false+) and is normalized the same
|
|
331
|
+
# way; see {build_session}.
|
|
283
332
|
# @return [Object] handler-specific identifier for the enqueued request
|
|
284
333
|
def ask(message = nil, context: {}, session: nil, callback: nil, **options)
|
|
285
|
-
session_options = options.slice(*
|
|
334
|
+
session_options = options.slice(*SESSION_OPTIONS)
|
|
286
335
|
raise ArgumentError.new("session options cannot be passed when a session is provided") if session && session_options.any?
|
|
287
336
|
|
|
288
337
|
session ||= build_session(**session_options)
|
|
289
338
|
session.user(message) if message
|
|
290
339
|
|
|
291
|
-
ask_options = options.except(*
|
|
340
|
+
ask_options = with_declared_stream(options.except(*SESSION_OPTIONS))
|
|
292
341
|
iterations = max_tool_iterations
|
|
293
342
|
ask_options[:max_tool_iterations] ||= iterations if iterations
|
|
294
343
|
|
|
@@ -371,7 +420,7 @@ module PatientLLM
|
|
|
371
420
|
# @param options [Hash] the same per-request overrides accepted by {ask}
|
|
372
421
|
# @return [PatientLLM::RequestPreview] the url, headers, and JSON payload
|
|
373
422
|
def preview_request(message = nil, session: nil, context: nil, callback: nil, **options)
|
|
374
|
-
session_options = options.slice(*
|
|
423
|
+
session_options = options.slice(*SESSION_OPTIONS)
|
|
375
424
|
raise ArgumentError.new("session options cannot be passed when a session is provided") if session && session_options.any?
|
|
376
425
|
|
|
377
426
|
session ||= build_session(**session_options)
|
|
@@ -380,7 +429,7 @@ module PatientLLM
|
|
|
380
429
|
PatientLLM.preview_request(
|
|
381
430
|
session,
|
|
382
431
|
provider: provider_name!,
|
|
383
|
-
**options.except(*
|
|
432
|
+
**with_declared_stream(options.except(*SESSION_OPTIONS))
|
|
384
433
|
)
|
|
385
434
|
end
|
|
386
435
|
|
|
@@ -388,10 +437,27 @@ module PatientLLM
|
|
|
388
437
|
# the caller take precedence over the agent's declarations for the same
|
|
389
438
|
# fields (pass an explicit nil to unset a declared value for one request).
|
|
390
439
|
#
|
|
440
|
+
# The reasoning: option accepts the same values as the {reasoning} DSL
|
|
441
|
+
# and is normalized the same way, so +reasoning: :low+,
|
|
442
|
+
# +reasoning: {budget_tokens: 2048}+, and +reasoning: false+ produce the
|
|
443
|
+
# same session reasoning as the equivalent declarations. An explicit
|
|
444
|
+
# +reasoning: nil+ sends no reasoning configuration for the request. A
|
|
445
|
+
# Hash with keys other than :effort, :budget_tokens, and :enabled is
|
|
446
|
+
# used as the session's raw reasoning configuration for provider-specific
|
|
447
|
+
# settings.
|
|
448
|
+
#
|
|
449
|
+
# @param options [Hash] PromptBuilder::Session::INITIALIZE_OPTIONS values
|
|
391
450
|
# @return [PromptBuilder::Session]
|
|
451
|
+
# @raise [ArgumentError] if the reasoning: option is invalid
|
|
392
452
|
def build_session(**options)
|
|
453
|
+
overridden = options.keys
|
|
454
|
+
if options.key?(:reasoning) && !raw_reasoning_configuration?(options[:reasoning])
|
|
455
|
+
request_reasoning = normalize_reasoning(options.delete(:reasoning))
|
|
456
|
+
end
|
|
457
|
+
|
|
393
458
|
session = PromptBuilder::Session.new(**options)
|
|
394
|
-
|
|
459
|
+
apply_reasoning(session, request_reasoning) if request_reasoning
|
|
460
|
+
apply_configuration(session, except: overridden)
|
|
395
461
|
session
|
|
396
462
|
end
|
|
397
463
|
|
|
@@ -410,13 +476,14 @@ module PatientLLM
|
|
|
410
476
|
temperature_value = temperature
|
|
411
477
|
max_output_tokens_value = max_output_tokens
|
|
412
478
|
extra_value = extra
|
|
479
|
+
reasoning_value = reasoning
|
|
413
480
|
apply_system_message(session, system_message) if system_message && !except.include?(:system)
|
|
414
481
|
session.model = model_value if model_value && !except.include?(:model)
|
|
415
482
|
session.instructions = instructions_value if instructions_value && !except.include?(:instructions)
|
|
416
483
|
session.temperature = temperature_value if temperature_value && !except.include?(:temperature)
|
|
417
484
|
session.max_output_tokens = max_output_tokens_value if max_output_tokens_value && !except.include?(:max_output_tokens)
|
|
418
485
|
session.extra = extra_value if extra_value && !except.include?(:extra)
|
|
419
|
-
session
|
|
486
|
+
apply_reasoning(session, reasoning_value) if reasoning_value && !except.include?(:reasoning)
|
|
420
487
|
if output_schema && !except.include?(:text)
|
|
421
488
|
session.json_output(output_schema[:schema], name: output_schema[:name], strict: output_schema[:strict])
|
|
422
489
|
end
|
|
@@ -445,6 +512,15 @@ module PatientLLM
|
|
|
445
512
|
provider || raise(ArgumentError, "#{self} must declare a provider")
|
|
446
513
|
end
|
|
447
514
|
|
|
515
|
+
# Add the agent's stream declaration to request options that do not set
|
|
516
|
+
# stream: themselves.
|
|
517
|
+
def with_declared_stream(options)
|
|
518
|
+
return options if options.key?(:stream)
|
|
519
|
+
|
|
520
|
+
declared = stream
|
|
521
|
+
declared.nil? ? options : options.merge(stream: declared)
|
|
522
|
+
end
|
|
523
|
+
|
|
448
524
|
# Resolve the :callback option to the class name that travels through the
|
|
449
525
|
# job queue. The class is resolved here so an unusable callback fails
|
|
450
526
|
# where ask was called rather than later in a worker process.
|
|
@@ -485,6 +561,58 @@ module PatientLLM
|
|
|
485
561
|
end
|
|
486
562
|
end
|
|
487
563
|
|
|
564
|
+
# Normalize a reasoning value into the `session.think` options returned
|
|
565
|
+
# by {reasoning}: nil for no configuration, +{enabled: false}+ (with an
|
|
566
|
+
# optional effort) for reasoning off, and +{effort:}+, +{budget_tokens:}+,
|
|
567
|
+
# or +{}+ (API defaults) for reasoning on. Already-normalized values pass
|
|
568
|
+
# through unchanged.
|
|
569
|
+
def normalize_reasoning(value, effort: nil, budget_tokens: nil)
|
|
570
|
+
case value
|
|
571
|
+
when nil
|
|
572
|
+
return nil if effort.nil? && budget_tokens.nil?
|
|
573
|
+
|
|
574
|
+
normalize_reasoning(true, effort: effort, budget_tokens: budget_tokens)
|
|
575
|
+
when true
|
|
576
|
+
raise ArgumentError, "pass a level, effort:, or budget_tokens: — not more than one" if effort && budget_tokens
|
|
577
|
+
|
|
578
|
+
{effort: effort&.to_s, budget_tokens: budget_tokens}.compact
|
|
579
|
+
when false
|
|
580
|
+
raise ArgumentError, "budget_tokens: cannot be combined with reasoning false" if budget_tokens
|
|
581
|
+
|
|
582
|
+
{enabled: false, effort: effort&.to_s}.compact
|
|
583
|
+
when Symbol, String
|
|
584
|
+
raise ArgumentError, "pass a level, effort:, or budget_tokens: — not more than one" if effort || budget_tokens
|
|
585
|
+
|
|
586
|
+
{effort: value.to_s}
|
|
587
|
+
when Hash
|
|
588
|
+
raise ArgumentError, "pass reasoning options as a Hash or as keywords, not both" if effort || budget_tokens
|
|
589
|
+
|
|
590
|
+
options = value.transform_keys(&:to_sym)
|
|
591
|
+
unknown = options.keys - REASONING_OPTIONS
|
|
592
|
+
raise ArgumentError, "unknown reasoning options: #{unknown.join(", ")}" unless unknown.empty?
|
|
593
|
+
|
|
594
|
+
enabled = options.fetch(:enabled, true)
|
|
595
|
+
raise ArgumentError, "reasoning enabled: must be true or false" unless enabled == true || enabled == false
|
|
596
|
+
|
|
597
|
+
normalize_reasoning(enabled, effort: options[:effort], budget_tokens: options[:budget_tokens])
|
|
598
|
+
else
|
|
599
|
+
raise ArgumentError, "reasoning must be an effort level, a Hash of options, true, false, or nil; got #{value.inspect}"
|
|
600
|
+
end
|
|
601
|
+
end
|
|
602
|
+
|
|
603
|
+
# Whether a per-request reasoning value is a provider-specific Hash to
|
|
604
|
+
# set on the session as-is rather than options to normalize.
|
|
605
|
+
def raw_reasoning_configuration?(value)
|
|
606
|
+
value.is_a?(Hash) && value.keys.any? { |key| !REASONING_OPTIONS.include?(key.to_sym) }
|
|
607
|
+
end
|
|
608
|
+
|
|
609
|
+
# Apply normalized reasoning options (see {normalize_reasoning}) to a
|
|
610
|
+
# session with `session.think`.
|
|
611
|
+
def apply_reasoning(session, options)
|
|
612
|
+
options = options.dup
|
|
613
|
+
session.think(options.delete(:enabled) != false, **options)
|
|
614
|
+
end
|
|
615
|
+
|
|
488
616
|
# Shared implementation of the scalar get-or-set DSL methods. With no
|
|
489
617
|
# argument and no block, reads the setting; when the stored value
|
|
490
618
|
# responds to #call it is called and the result used, so values can be
|
data/lib/patient_llm/callback.rb
CHANGED
|
@@ -17,6 +17,12 @@ module PatientLLM
|
|
|
17
17
|
# The user callback receives a `PromptBuilder::Response` object. Access
|
|
18
18
|
# the response text via `response.text`, token usage via `response.usage`,
|
|
19
19
|
# and model id via `response.model`.
|
|
20
|
+
#
|
|
21
|
+
# A streamed response body is reassembled into the same response the
|
|
22
|
+
# non-streaming endpoint would have returned. A stream that reports an error
|
|
23
|
+
# event, ends before its terminal event, or is malformed is routed to the
|
|
24
|
+
# user callback's on_error method as a `PatientHttp::RequestError` with the
|
|
25
|
+
# error type :stream_error.
|
|
20
26
|
class Callback
|
|
21
27
|
# Default maximum number of tool-execution rounds before the loop stops and
|
|
22
28
|
# routes a `PatientHttp::RequestError` (error type :max_tool_iterations) to
|
|
@@ -87,9 +93,21 @@ module PatientLLM
|
|
|
87
93
|
prepare_user_callback(user_callback, session: session, provider: provider_name, callback_args: callback_args, http_response: response, request_id: original_request_id)
|
|
88
94
|
|
|
89
95
|
serializer = resolve_serializer(callback_args)
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
96
|
+
if streamed_response?(response, callback_args)
|
|
97
|
+
begin
|
|
98
|
+
# A response with no body (such as a 204) is an empty stream, which
|
|
99
|
+
# the parser reports as incomplete.
|
|
100
|
+
llm_response = PromptBuilder::Response.parse_stream(response.body || "", serializer, headers: response.headers)
|
|
101
|
+
rescue PromptBuilder::UnexpectedPayloadError, PromptBuilder::ParseError => e
|
|
102
|
+
error = request_error(e, :stream_error, response, original_request_id)
|
|
103
|
+
invoke_user_callback(user_callback, :on_error, session: session, provider: provider_name, callback_args: user_callback_args(callback_args), error: error, http_response: response, request_id: original_request_id)
|
|
104
|
+
return
|
|
105
|
+
end
|
|
106
|
+
else
|
|
107
|
+
# The headers let formats that return response metadata in HTTP headers
|
|
108
|
+
# (e.g. the Converse response id) populate the parsed response.
|
|
109
|
+
llm_response = PromptBuilder::Response.parse(response.json, serializer, headers: response.headers)
|
|
110
|
+
end
|
|
93
111
|
|
|
94
112
|
if should_auto_execute_tools?(llm_response, user_callback)
|
|
95
113
|
continue_tool_loop(session, provider_name, llm_response, callback_args, response, user_callback, original_request_id)
|
|
@@ -192,6 +210,18 @@ module PatientLLM
|
|
|
192
210
|
provider_config&.dig(:serializer) || :chat_completion
|
|
193
211
|
end
|
|
194
212
|
|
|
213
|
+
# Whether the response body is a stream. The stream flag recorded at
|
|
214
|
+
# dispatch covers requests streamed by request option, stream param,
|
|
215
|
+
# provider setting, or session field; the content type also catches an
|
|
216
|
+
# endpoint that streams without being asked, such as a custom path to a
|
|
217
|
+
# streaming endpoint.
|
|
218
|
+
def streamed_response?(response, callback_args)
|
|
219
|
+
return true if callback_args.fetch(:stream, nil) == true
|
|
220
|
+
|
|
221
|
+
content_type = response.headers["content-type"].to_s.downcase
|
|
222
|
+
STREAM_CONTENT_TYPES.any? { |type| content_type.start_with?(type) }
|
|
223
|
+
end
|
|
224
|
+
|
|
195
225
|
def max_tool_iterations(callback_args)
|
|
196
226
|
callback_args.fetch(:max_tool_iterations, MAX_TOOL_ITERATIONS).to_i
|
|
197
227
|
end
|
|
@@ -274,11 +304,17 @@ module PatientLLM
|
|
|
274
304
|
def max_tool_iterations_error(callback_args, http_response, request_id)
|
|
275
305
|
limit = max_tool_iterations(callback_args)
|
|
276
306
|
exception = MaxToolIterationsError.new("Tool-call loop exceeded #{limit} iterations")
|
|
307
|
+
request_error(exception, :max_tool_iterations, http_response, request_id)
|
|
308
|
+
end
|
|
309
|
+
|
|
310
|
+
# Wrap an exception raised while handling a response in a RequestError so
|
|
311
|
+
# it reaches the user callback's on_error method like a transport error.
|
|
312
|
+
def request_error(exception, error_type, http_response, request_id)
|
|
277
313
|
PatientHttp::RequestError.new(
|
|
278
314
|
class_name: exception.class.name,
|
|
279
315
|
message: exception.message,
|
|
280
|
-
backtrace: [],
|
|
281
|
-
error_type:
|
|
316
|
+
backtrace: exception.backtrace || [],
|
|
317
|
+
error_type: error_type,
|
|
282
318
|
duration: http_response&.duration,
|
|
283
319
|
request_id: request_id,
|
|
284
320
|
url: http_response&.url,
|
|
@@ -28,9 +28,18 @@ module PatientLLM
|
|
|
28
28
|
|
|
29
29
|
attr_reader :session_offload_options
|
|
30
30
|
|
|
31
|
+
# Name of the PatientHttp processor that executes LLM requests, used when a
|
|
32
|
+
# request does not pass its own processor: option. The value can be a
|
|
33
|
+
# Symbol, a String, or a callable resolved on every request. When nil, the
|
|
34
|
+
# default processor is used.
|
|
35
|
+
#
|
|
36
|
+
# @return [Symbol, String, #call, nil]
|
|
37
|
+
attr_accessor :processor
|
|
38
|
+
|
|
31
39
|
def initialize
|
|
32
40
|
@providers = {}
|
|
33
41
|
@session_offload_options = nil
|
|
42
|
+
@processor = nil
|
|
34
43
|
end
|
|
35
44
|
|
|
36
45
|
# Register a provider.
|
|
@@ -60,11 +69,18 @@ module PatientLLM
|
|
|
60
69
|
# @param preprocessors [String, Symbol, Array<String, Symbol>, #call, nil] Names of request
|
|
61
70
|
# preprocessors to apply to every request (e.g. for request signing). Names must be
|
|
62
71
|
# registered on the PatientHttp configuration with `register_preprocessor`.
|
|
63
|
-
# @param timeout [Numeric, #call, nil] Request timeout in seconds for this provider's requests
|
|
72
|
+
# @param timeout [Numeric, #call, nil] Request timeout in seconds for this provider's requests.
|
|
73
|
+
# For a streamed request this limits the full stream.
|
|
74
|
+
# @param stream [Boolean, #call, nil] Whether to stream this provider's responses. The
|
|
75
|
+
# streamed body is reassembled into a single response before the callback runs.
|
|
76
|
+
# @param read_timeout [Numeric, #call, nil] Maximum seconds to wait for data from the
|
|
77
|
+
# server on a streamed request. Not applied to requests that are not streamed.
|
|
78
|
+
# @param max_response_size [Integer, #call, nil] Maximum response body size in bytes
|
|
79
|
+
# for this provider's requests. A streamed body is larger than the equivalent JSON body.
|
|
64
80
|
# @param max_tool_iterations [Integer, #call, nil] Maximum automatic tool-execution rounds
|
|
65
81
|
# for this provider's requests (default: PatientLLM::Callback::MAX_TOOL_ITERATIONS)
|
|
66
82
|
# @return [void]
|
|
67
|
-
def provider(name, preset: nil, url: nil, api_key: nil, region: nil, headers: nil, serializer: nil, path: nil, params: {}, preprocessors: nil, timeout: nil, max_tool_iterations: nil)
|
|
83
|
+
def provider(name, preset: nil, url: nil, api_key: nil, region: nil, headers: nil, serializer: nil, path: nil, params: {}, preprocessors: nil, timeout: nil, stream: nil, read_timeout: nil, max_response_size: nil, max_tool_iterations: nil)
|
|
68
84
|
preset_config = preset ? Presets.fetch(preset) : nil
|
|
69
85
|
options = {
|
|
70
86
|
preset_config: preset_config,
|
|
@@ -76,6 +92,9 @@ module PatientLLM
|
|
|
76
92
|
params: params,
|
|
77
93
|
preprocessors: preprocessors,
|
|
78
94
|
timeout: timeout,
|
|
95
|
+
stream: stream,
|
|
96
|
+
read_timeout: read_timeout,
|
|
97
|
+
max_response_size: max_response_size,
|
|
79
98
|
max_tool_iterations: max_tool_iterations
|
|
80
99
|
}
|
|
81
100
|
|
|
@@ -158,6 +177,9 @@ module PatientLLM
|
|
|
158
177
|
params: options[:params] || {},
|
|
159
178
|
preprocessors: options[:preprocessors],
|
|
160
179
|
timeout: options[:timeout],
|
|
180
|
+
stream: options[:stream],
|
|
181
|
+
read_timeout: options[:read_timeout],
|
|
182
|
+
max_response_size: options[:max_response_size],
|
|
161
183
|
max_tool_iterations: options[:max_tool_iterations]
|
|
162
184
|
}
|
|
163
185
|
end
|
data/lib/patient_llm.rb
CHANGED
|
@@ -31,6 +31,18 @@ module PatientLLM
|
|
|
31
31
|
gemini: "v1beta/models/{model}:generateContent"
|
|
32
32
|
}.freeze
|
|
33
33
|
|
|
34
|
+
# Default API paths per serializer format for streamed requests. Gemini and
|
|
35
|
+
# Bedrock Converse select streaming by endpoint; the other formats stream
|
|
36
|
+
# from the same endpoint with `stream: true` in the request body.
|
|
37
|
+
STREAMING_SERIALIZER_PATHS = SERIALIZER_PATHS.merge(
|
|
38
|
+
converse: "model/{model}/converse-stream",
|
|
39
|
+
gemini: "v1beta/models/{model}:streamGenerateContent?alt=sse"
|
|
40
|
+
).freeze
|
|
41
|
+
|
|
42
|
+
# Content types of streamed response bodies: Server-Sent Events, and the AWS
|
|
43
|
+
# event-stream framing used by Bedrock Converse.
|
|
44
|
+
STREAM_CONTENT_TYPES = ["text/event-stream", "application/vnd.amazon.eventstream"].freeze
|
|
45
|
+
|
|
34
46
|
# Required version header for the Anthropic Messages API.
|
|
35
47
|
ANTHROPIC_VERSION = "2023-06-01"
|
|
36
48
|
|
|
@@ -153,11 +165,23 @@ module PatientLLM
|
|
|
153
165
|
# preprocessors; pass an empty array to clear them for this request.
|
|
154
166
|
# @param timeout [Numeric, nil] Request timeout in seconds for this request. Overrides
|
|
155
167
|
# the provider's timeout; defaults to the PatientHttp processor configuration.
|
|
168
|
+
# For a streamed request this limits the full stream.
|
|
169
|
+
# @param stream [Boolean, nil] Whether to stream the response. The streamed body is
|
|
170
|
+
# reassembled into a single response before the callback runs. Overrides the
|
|
171
|
+
# provider's setting; when neither is set, the session's `stream` field applies.
|
|
172
|
+
# @param read_timeout [Numeric, nil] Maximum seconds to wait for data from the
|
|
173
|
+
# server on a streamed request. Overrides the provider's read timeout. Not
|
|
174
|
+
# applied to requests that are not streamed.
|
|
175
|
+
# @param max_response_size [Integer, nil] Maximum response body size in bytes.
|
|
176
|
+
# Overrides the provider's setting; defaults to the PatientHttp configuration.
|
|
156
177
|
# @param max_tool_iterations [Integer, nil] Maximum automatic tool-execution rounds
|
|
157
178
|
# for this request. Overrides the provider's setting; defaults to
|
|
158
179
|
# {Callback::MAX_TOOL_ITERATIONS}.
|
|
180
|
+
# @param processor [String, Symbol, nil] Name of the PatientHttp processor that
|
|
181
|
+
# should execute this request. Overrides the configuration's processor;
|
|
182
|
+
# defaults to nil, which uses the default processor.
|
|
159
183
|
# @return [Object] Handler-specific identifier for the enqueued request
|
|
160
|
-
def ask(session, provider:, callback:, callback_args: {}, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, max_tool_iterations: nil)
|
|
184
|
+
def ask(session, provider:, callback:, callback_args: {}, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, stream: nil, read_timeout: nil, max_response_size: nil, max_tool_iterations: nil, processor: nil)
|
|
161
185
|
request_options = build_request_options(
|
|
162
186
|
url: url,
|
|
163
187
|
serializer: serializer,
|
|
@@ -166,7 +190,11 @@ module PatientLLM
|
|
|
166
190
|
params: params,
|
|
167
191
|
preprocessors: preprocessors,
|
|
168
192
|
timeout: timeout,
|
|
169
|
-
|
|
193
|
+
stream: stream,
|
|
194
|
+
read_timeout: read_timeout,
|
|
195
|
+
max_response_size: max_response_size,
|
|
196
|
+
max_tool_iterations: max_tool_iterations,
|
|
197
|
+
processor: processor
|
|
170
198
|
)
|
|
171
199
|
|
|
172
200
|
dispatch(session, provider: provider, callback: callback, callback_args: callback_args, request_options: request_options)
|
|
@@ -193,10 +221,18 @@ module PatientLLM
|
|
|
193
221
|
# @param preprocessors [String, Symbol, Array<String, Symbol>, nil] Accepted for
|
|
194
222
|
# parity with {.ask}; preprocessors are not applied to the preview.
|
|
195
223
|
# @param timeout [Numeric, nil] Accepted for parity with {.ask}; does not affect the preview.
|
|
224
|
+
# @param stream [Boolean, nil] Whether to stream the response. Streaming changes the
|
|
225
|
+
# endpoint path for some formats and the payload, so it is reflected in the preview.
|
|
226
|
+
# @param read_timeout [Numeric, nil] Accepted for parity with {.ask}; does not affect
|
|
227
|
+
# the preview.
|
|
228
|
+
# @param max_response_size [Integer, nil] Accepted for parity with {.ask}; does not
|
|
229
|
+
# affect the preview.
|
|
196
230
|
# @param max_tool_iterations [Integer, nil] Accepted for parity with {.ask}; does not
|
|
197
231
|
# affect the preview.
|
|
232
|
+
# @param processor [String, Symbol, nil] Accepted for parity with {.ask}; does not
|
|
233
|
+
# affect the preview.
|
|
198
234
|
# @return [RequestPreview] The url, headers, and JSON payload the request would send
|
|
199
|
-
def preview_request(session, provider:, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, max_tool_iterations: nil)
|
|
235
|
+
def preview_request(session, provider:, url: nil, serializer: nil, path: nil, headers: nil, params: nil, preprocessors: nil, timeout: nil, stream: nil, read_timeout: nil, max_response_size: nil, max_tool_iterations: nil, processor: nil)
|
|
200
236
|
request_options = build_request_options(
|
|
201
237
|
url: url,
|
|
202
238
|
serializer: serializer,
|
|
@@ -205,7 +241,11 @@ module PatientLLM
|
|
|
205
241
|
params: params,
|
|
206
242
|
preprocessors: preprocessors,
|
|
207
243
|
timeout: timeout,
|
|
208
|
-
|
|
244
|
+
stream: stream,
|
|
245
|
+
read_timeout: read_timeout,
|
|
246
|
+
max_response_size: max_response_size,
|
|
247
|
+
max_tool_iterations: max_tool_iterations,
|
|
248
|
+
processor: processor
|
|
209
249
|
)
|
|
210
250
|
|
|
211
251
|
resolved = resolve_request(session, self.provider(provider) || {}, request_options)
|
|
@@ -238,6 +278,7 @@ module PatientLLM
|
|
|
238
278
|
callback: callback.to_s,
|
|
239
279
|
custom: PromptBuilder.jsonify(callback_args || {}),
|
|
240
280
|
request_options: request_options,
|
|
281
|
+
stream: resolved.stream,
|
|
241
282
|
max_tool_iterations: resolved.max_tool_iterations,
|
|
242
283
|
tool_iteration: tool_iteration,
|
|
243
284
|
original_request_id: original_request_id
|
|
@@ -250,7 +291,10 @@ module PatientLLM
|
|
|
250
291
|
json: resolved.payload,
|
|
251
292
|
headers: resolved.headers,
|
|
252
293
|
preprocessors: resolved.preprocessors,
|
|
253
|
-
timeout: resolved.timeout
|
|
294
|
+
timeout: resolved.timeout,
|
|
295
|
+
read_timeout: resolved.read_timeout,
|
|
296
|
+
max_response_size: resolved.max_response_size,
|
|
297
|
+
processor: resolved.processor
|
|
254
298
|
)
|
|
255
299
|
PatientHttp.execute_inline(
|
|
256
300
|
request: request,
|
|
@@ -265,6 +309,9 @@ module PatientLLM
|
|
|
265
309
|
headers: resolved.headers,
|
|
266
310
|
preprocessors: resolved.preprocessors,
|
|
267
311
|
timeout: resolved.timeout,
|
|
312
|
+
read_timeout: resolved.read_timeout,
|
|
313
|
+
max_response_size: resolved.max_response_size,
|
|
314
|
+
processor: resolved.processor,
|
|
268
315
|
raise_error_responses: true,
|
|
269
316
|
callback: PatientLLM::Callback,
|
|
270
317
|
callback_args: dispatch_callback_args
|
|
@@ -276,14 +323,14 @@ module PatientLLM
|
|
|
276
323
|
|
|
277
324
|
# Fully resolved request produced by merging per-request options over the
|
|
278
325
|
# provider configuration.
|
|
279
|
-
ResolvedRequest = Data.define(:url, :serializer, :headers, :payload, :preprocessors, :timeout, :max_tool_iterations)
|
|
326
|
+
ResolvedRequest = Data.define(:url, :serializer, :headers, :payload, :preprocessors, :timeout, :stream, :read_timeout, :max_response_size, :max_tool_iterations, :processor)
|
|
280
327
|
private_constant :ResolvedRequest
|
|
281
328
|
|
|
282
329
|
# Normalize per-request overrides into a request options hash. The request
|
|
283
330
|
# options travel through the job queue in the callback args, which only
|
|
284
331
|
# permit JSON-native values; convert Symbols (e.g. serializer names,
|
|
285
332
|
# preprocessor names, header/param values) to Strings up front.
|
|
286
|
-
def build_request_options(url:, serializer:, path:, headers:, params:, preprocessors:, timeout:, max_tool_iterations:)
|
|
333
|
+
def build_request_options(url:, serializer:, path:, headers:, params:, preprocessors:, timeout:, stream:, read_timeout:, max_response_size:, max_tool_iterations:, processor:)
|
|
287
334
|
request_options = {}
|
|
288
335
|
request_options["url"] = url if url
|
|
289
336
|
request_options["serializer"] = serializer.to_s if serializer
|
|
@@ -292,7 +339,12 @@ module PatientLLM
|
|
|
292
339
|
request_options["params"] = params if params && !params.empty?
|
|
293
340
|
request_options["preprocessors"] = preprocessors if preprocessors
|
|
294
341
|
request_options["timeout"] = timeout if timeout
|
|
342
|
+
# An explicit false is kept so a request can turn off a provider's streaming.
|
|
343
|
+
request_options["stream"] = stream unless stream.nil?
|
|
344
|
+
request_options["read_timeout"] = read_timeout if read_timeout
|
|
345
|
+
request_options["max_response_size"] = max_response_size if max_response_size
|
|
295
346
|
request_options["max_tool_iterations"] = max_tool_iterations if max_tool_iterations
|
|
347
|
+
request_options["processor"] = processor.to_s if processor
|
|
296
348
|
|
|
297
349
|
PromptBuilder.jsonify(request_options)
|
|
298
350
|
end
|
|
@@ -308,7 +360,11 @@ module PatientLLM
|
|
|
308
360
|
resolved_serializer = (request_options["serializer"] || provider_config[:serializer] || :chat_completion).to_sym
|
|
309
361
|
validate_serializer!(resolved_serializer)
|
|
310
362
|
|
|
311
|
-
|
|
363
|
+
provider_params = deep_stringify_keys(provider_config[:params])
|
|
364
|
+
request_params = deep_stringify_keys(request_options["params"])
|
|
365
|
+
stream = resolve_stream(session, provider_config, request_options, provider_params, request_params)
|
|
366
|
+
default_paths = stream ? STREAMING_SERIALIZER_PATHS : SERIALIZER_PATHS
|
|
367
|
+
resolved_path = request_options["path"] || provider_config[:path] || default_paths[resolved_serializer]
|
|
312
368
|
if resolved_path.include?("{model}")
|
|
313
369
|
raise ArgumentError, "The endpoint path #{resolved_path.inspect} includes a {model} placeholder but session.model is not set" if session.model.nil?
|
|
314
370
|
|
|
@@ -323,10 +379,13 @@ module PatientLLM
|
|
|
323
379
|
resolved_headers = {"anthropic-version" => ANTHROPIC_VERSION}.merge(resolved_headers)
|
|
324
380
|
end
|
|
325
381
|
|
|
326
|
-
|
|
382
|
+
# The stream param is resolved with the stream option above and set
|
|
383
|
+
# through the serializer, which adds the options streaming needs (such
|
|
384
|
+
# as the Chat Completions usage option), so it is not merged on top.
|
|
385
|
+
resolved_params = provider_params.merge(request_params).except("stream")
|
|
327
386
|
|
|
328
|
-
payload = session
|
|
329
|
-
payload = deep_merge(payload,
|
|
387
|
+
payload = streaming_payload(session, resolved_serializer, stream)
|
|
388
|
+
payload = deep_merge(payload, resolved_params) unless resolved_params.empty?
|
|
330
389
|
|
|
331
390
|
ResolvedRequest.new(
|
|
332
391
|
url: join_url(resolved_url, resolved_path),
|
|
@@ -335,10 +394,53 @@ module PatientLLM
|
|
|
335
394
|
payload: payload,
|
|
336
395
|
preprocessors: request_options["preprocessors"] || provider_config[:preprocessors],
|
|
337
396
|
timeout: request_options["timeout"] || provider_config[:timeout],
|
|
338
|
-
|
|
397
|
+
stream: stream,
|
|
398
|
+
# The read timeout covers the wait for the response headers, which a
|
|
399
|
+
# non-streamed LLM request spends generating the whole response, so it
|
|
400
|
+
# only applies to streamed requests.
|
|
401
|
+
read_timeout: stream ? (request_options["read_timeout"] || provider_config[:read_timeout]) : nil,
|
|
402
|
+
max_response_size: request_options["max_response_size"] || provider_config[:max_response_size],
|
|
403
|
+
max_tool_iterations: (request_options["max_tool_iterations"] || provider_config[:max_tool_iterations] || Callback::MAX_TOOL_ITERATIONS).to_i,
|
|
404
|
+
processor: request_options["processor"] || configured_processor
|
|
339
405
|
)
|
|
340
406
|
end
|
|
341
407
|
|
|
408
|
+
# Whether the request streams its response. The first value that is set
|
|
409
|
+
# wins: the request option, a stream param on the request, the provider
|
|
410
|
+
# setting, a stream param on the provider, then the session's stream field.
|
|
411
|
+
def resolve_stream(session, provider_config, request_options, provider_params, request_params)
|
|
412
|
+
values = [
|
|
413
|
+
request_options["stream"],
|
|
414
|
+
request_params["stream"],
|
|
415
|
+
provider_config[:stream],
|
|
416
|
+
provider_params["stream"],
|
|
417
|
+
session.stream
|
|
418
|
+
]
|
|
419
|
+
values.find { |value| !value.nil? } == true
|
|
420
|
+
end
|
|
421
|
+
|
|
422
|
+
# Serialize the session with its stream field set to match the resolved
|
|
423
|
+
# streaming setting. The field is set on a shallow copy, so the setting
|
|
424
|
+
# does not become part of the conversation state and concurrent requests
|
|
425
|
+
# sharing the session do not see each other's setting. A session that never
|
|
426
|
+
# set the field keeps it unset when the request is not streamed.
|
|
427
|
+
def streaming_payload(session, serializer, stream)
|
|
428
|
+
value = stream || (session.stream.nil? ? nil : false)
|
|
429
|
+
return session.request_payload(serializer) if value == session.stream
|
|
430
|
+
|
|
431
|
+
copy = session.dup
|
|
432
|
+
copy.stream = value
|
|
433
|
+
copy.request_payload(serializer)
|
|
434
|
+
end
|
|
435
|
+
|
|
436
|
+
# The processor name from the configuration, calling a callable value and
|
|
437
|
+
# normalizing the result to a String. Nil when no processor is configured.
|
|
438
|
+
def configured_processor
|
|
439
|
+
value = @configuration&.processor
|
|
440
|
+
value = value.call if value.respond_to?(:call)
|
|
441
|
+
value&.to_s
|
|
442
|
+
end
|
|
443
|
+
|
|
342
444
|
# Replace secret reference header values with placeholders so a preview
|
|
343
445
|
# never resolves or exposes secret values.
|
|
344
446
|
def redact_secret_headers(headers)
|
data/patient_llm.gemspec
CHANGED
|
@@ -35,8 +35,8 @@ Gem::Specification.new do |spec|
|
|
|
35
35
|
|
|
36
36
|
spec.require_paths = ["lib"]
|
|
37
37
|
|
|
38
|
-
spec.required_ruby_version = ">= 3.
|
|
38
|
+
spec.required_ruby_version = ">= 3.3"
|
|
39
39
|
|
|
40
|
-
spec.add_dependency "patient_http", "
|
|
41
|
-
spec.add_dependency "prompt_builder", "~> 0.
|
|
40
|
+
spec.add_dependency "patient_http", ">= 1.8.0"
|
|
41
|
+
spec.add_dependency "prompt_builder", "~> 0.5"
|
|
42
42
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: patient_llm
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.10.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Brian Durand
|
|
@@ -13,30 +13,30 @@ dependencies:
|
|
|
13
13
|
name: patient_http
|
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
|
15
15
|
requirements:
|
|
16
|
-
- - "
|
|
16
|
+
- - ">="
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version:
|
|
18
|
+
version: 1.8.0
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
|
-
- - "
|
|
23
|
+
- - ">="
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version:
|
|
25
|
+
version: 1.8.0
|
|
26
26
|
- !ruby/object:Gem::Dependency
|
|
27
27
|
name: prompt_builder
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
29
29
|
requirements:
|
|
30
30
|
- - "~>"
|
|
31
31
|
- !ruby/object:Gem::Version
|
|
32
|
-
version: '0.
|
|
32
|
+
version: '0.5'
|
|
33
33
|
type: :runtime
|
|
34
34
|
prerelease: false
|
|
35
35
|
version_requirements: !ruby/object:Gem::Requirement
|
|
36
36
|
requirements:
|
|
37
37
|
- - "~>"
|
|
38
38
|
- !ruby/object:Gem::Version
|
|
39
|
-
version: '0.
|
|
39
|
+
version: '0.5'
|
|
40
40
|
email:
|
|
41
41
|
- bbdurand@gmail.com
|
|
42
42
|
executables: []
|
|
@@ -75,7 +75,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
75
75
|
requirements:
|
|
76
76
|
- - ">="
|
|
77
77
|
- !ruby/object:Gem::Version
|
|
78
|
-
version: '3.
|
|
78
|
+
version: '3.3'
|
|
79
79
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
80
80
|
requirements:
|
|
81
81
|
- - ">="
|