phronomy 0.29.0 → 0.30.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +107 -1
- data/README.md +17 -6
- data/docs/architecture/agent-configuration-and-tool-binding.md +8 -0
- data/docs/architecture/agent-iteration-budget.md +81 -0
- data/docs/architecture/dx05-agent-scope.md +9 -2
- data/docs/architecture/error-api-review.md +4 -4
- data/docs/architecture/generator-verifier-lifecycle-result.md +126 -0
- data/docs/architecture/generator-verifier-ownership.md +18 -109
- data/docs/architecture/llm-continuation-metadata.md +88 -0
- data/docs/architecture/lm-studio-output-failures.md +111 -0
- data/docs/architecture/r8-unit1.md +68 -68
- data/docs/architecture/r8-unit15.md +3 -0
- data/docs/architecture/r8-unit2.md +34 -34
- data/docs/architecture/r8-unit7.md +2 -1
- data/docs/architecture/structured-output-and-scope.md +183 -0
- data/docs/architecture/tracing.md +22 -4
- data/docs/architecture/transport-error-boundaries.md +103 -0
- data/docs/architecture/trust-acceptance-followup.md +87 -0
- data/docs/decisions/029-semantic-completion-and-application-effect-boundary.md +1 -1
- data/docs/design/durable-semantic-coordination/CHANGELOG_V2_REVISION_2.md +23 -23
- data/docs/design/durable-semantic-coordination/IMPLEMENTATION_REPORT.md +99 -99
- data/docs/design/durable-semantic-coordination/RECOVERY_CONTRACT_CLARIFICATIONS.md +135 -136
- data/docs/features.md +10 -1
- data/docs/migrations/r8-unit15.md +3 -0
- data/gemfiles/http_minimum.gemfile +6 -1
- data/lib/phronomy/agent/admission.rb +1 -1
- data/lib/phronomy/agent/async_event_api.rb +16 -7
- data/lib/phronomy/agent/base.rb +8 -1
- data/lib/phronomy/agent/context_importer.rb +4 -0
- data/lib/phronomy/agent/execution/agent_invocation.rb +6 -2
- data/lib/phronomy/agent/execution/dispatch_preparation.rb +6 -1
- data/lib/phronomy/agent/execution/exact_execution.rb +9 -0
- data/lib/phronomy/agent/execution/execution_cancellation.rb +1 -1
- data/lib/phronomy/agent/execution/execution_coordinator.rb +63 -5
- data/lib/phronomy/agent/execution/execution_outcome_committer.rb +24 -4
- data/lib/phronomy/agent/execution/execution_registry.rb +5 -1
- data/lib/phronomy/agent/execution/initial_preparation.rb +10 -1
- data/lib/phronomy/agent/execution/invocation_actions.rb +1 -1
- data/lib/phronomy/agent/execution/invocation_transitions.rb +4 -4
- data/lib/phronomy/agent/execution/iteration_budget.rb +35 -0
- data/lib/phronomy/agent/execution/runtime_record_encoder.rb +3 -0
- data/lib/phronomy/agent/execution_observation.rb +5 -5
- data/lib/phronomy/agent/lifecycle/scope.rb +33 -23
- data/lib/phronomy/agent/lifecycle/scope_cleanup_error.rb +4 -2
- data/lib/phronomy/agent/output_contract.rb +64 -0
- data/lib/phronomy/agent/output_error.rb +53 -0
- data/lib/phronomy/agent/recovery/recovery_coordinator/continuation.rb +3 -3
- data/lib/phronomy/agent/recovery/recovery_coordinator/installation.rb +1 -0
- data/lib/phronomy/agent/recovery/recovery_coordinator.rb +1 -1
- data/lib/phronomy/agent/recovery/recovery_support.rb +5 -1
- data/lib/phronomy/agent/runtime_binding/engine_session_builder.rb +2 -1
- data/lib/phronomy/agent/runtime_input.rb +15 -4
- data/lib/phronomy/agent/saved_context_reader.rb +2 -1
- data/lib/phronomy/agent/store.rb +10 -2
- data/lib/phronomy/common/values/immutable.rb +33 -0
- data/lib/phronomy/context/assembly.rb +7 -4
- data/lib/phronomy/embeddings/backends/ruby_llm_embeddings.rb +3 -1
- data/lib/phronomy/engine/fsm_session.rb +16 -1
- data/lib/phronomy/generation/generator_verifier/agent_operation.rb +28 -5
- data/lib/phronomy/generation/generator_verifier/agent_result_receiver.rb +14 -27
- data/lib/phronomy/generation/generator_verifier/approval.rb +30 -0
- data/lib/phronomy/generation/generator_verifier/cleanup_error.rb +15 -0
- data/lib/phronomy/generation/generator_verifier/invocation.rb +147 -0
- data/lib/phronomy/generation/generator_verifier/pipeline_state.rb +16 -0
- data/lib/phronomy/generation/generator_verifier/protocol.rb +55 -0
- data/lib/phronomy/generation/generator_verifier/result.rb +121 -0
- data/lib/phronomy/generation/generator_verifier/unaccepted_result_error.rb +14 -0
- data/lib/phronomy/generation/generator_verifier/workflow_builder.rb +13 -11
- data/lib/phronomy/generation/generator_verifier.rb +170 -85
- data/lib/phronomy/llm_adapter/backends/lm_studio.rb +139 -0
- data/lib/phronomy/llm_adapter/backends/ruby_llm.rb +134 -12
- data/lib/phronomy/llm_adapter/backends/structured_response_capture.rb +41 -0
- data/lib/phronomy/llm_adapter/message.rb +4 -0
- data/lib/phronomy/llm_adapter/request.rb +17 -2
- data/lib/phronomy/llm_adapter/response.rb +11 -4
- data/lib/phronomy/llm_adapter/structured_output_unsupported_error.rb +14 -0
- data/lib/phronomy/llm_adapter/transport_error.rb +4 -0
- data/lib/phronomy/multi_agent/shared_state.rb +1 -0
- data/lib/phronomy/output_parser/json_schema_parser.rb +113 -0
- data/lib/phronomy/output_parser/parse_error.rb +10 -1
- data/lib/phronomy/runtime_composition/agent_defaults.rb +4 -0
- data/lib/phronomy/tracing/automatic.rb +14 -8
- data/lib/phronomy/tracing/base.rb +2 -1
- data/lib/phronomy/tracing/langfuse_tracer.rb +3 -3
- data/lib/phronomy/tracing/observation.rb +2 -1
- data/lib/phronomy/tracing/open_telemetry_tracer.rb +3 -3
- data/lib/phronomy/version.rb +1 -1
- data/lib/phronomy.rb +1 -2
- data/scripts/api_snapshot.rb +3 -0
- data/scripts/migrate_generator_verifier_result.rb +18 -0
- data/sig/phronomy/agent/execution_contract.rbs +4 -3
- data/sig/phronomy/agent.rbs +20 -7
- data/sig/phronomy/extensions.rbs +8 -3
- data/sig/phronomy/generation/generator_verifier.rbs +70 -0
- data/sig/phronomy/llm_adapter/backends/lm_studio.rbs +8 -0
- data/sig/phronomy/llm_adapter/backends/ruby_llm.rbs +1 -0
- data/sig/phronomy/llm_adapter/values.rbs +8 -2
- data/sig/phronomy/tracing/observation.rbs +1 -1
- data/sig/phronomy.rbs +2 -0
- metadata +59 -8
- data/lib/phronomy/generation/generator_verifier/default_parser.rb +0 -27
- data/lib/phronomy/generation/low_confidence_error.rb +0 -14
- data/lib/phronomy/runtime_composition/generation_defaults.rb +0 -6
- data/sig/phronomy/generation/low_confidence_error.rbs +0 -6
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
# Structured Agent output and application lifetimes
|
|
2
|
+
|
|
3
|
+
This contract supersedes the parser fallback described in the earlier Generation
|
|
4
|
+
ownership documents. Agent owns the promise made by `output_schema:`; Generation
|
|
5
|
+
owns the draft/review protocol and evaluation; the application owns publication,
|
|
6
|
+
authorization and any decision to start another attempt.
|
|
7
|
+
|
|
8
|
+
## Public behavior
|
|
9
|
+
|
|
10
|
+
`Agent::Base#invoke`, `invoke_async`, `stream`, `stream_async` and
|
|
11
|
+
`start_reserved_async` accept `output_schema: nil`. `run_once` forwards this option.
|
|
12
|
+
Without a schema the existing free-text behavior remains. With a schema, successful
|
|
13
|
+
`output` and `result` are deeply frozen JSON objects with String keys; `raw_output`
|
|
14
|
+
preserves the model text. Token events are provisional, not validated objects.
|
|
15
|
+
The final Task and terminal event are produced only after validation. Output
|
|
16
|
+
filters run before validation and cannot silently break the promised shape.
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
schema = {
|
|
20
|
+
type: "object", properties: {answer: {type: "string"}},
|
|
21
|
+
required: ["answer"], additionalProperties: false
|
|
22
|
+
}
|
|
23
|
+
value = Phronomy::Agent.open_scope do |scope|
|
|
24
|
+
agent = scope.create(AnswerAgent)
|
|
25
|
+
agent.invoke_async("Explain the result", output_schema: schema)
|
|
26
|
+
end
|
|
27
|
+
puts value.fetch(:output).fetch("answer")
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
C17 `OutputParser::JsonSchemaParser.new(schema:).parse(text)` performs strict JSON
|
|
31
|
+
decoding and JSON Schema validation. It does not extract fenced JSON, remove model
|
|
32
|
+
control tokens, coerce numbers, clamp scores or repair missing fields. Duplicate
|
|
33
|
+
keys, comments, invalid escapes, control characters and nonfinite values fail.
|
|
34
|
+
Existing extraction-oriented parsers remain available for their separate purposes;
|
|
35
|
+
they do not define Agent's structured-output guarantee.
|
|
36
|
+
|
|
37
|
+
The supported vocabulary is `$schema`, `$defs`, `$ref`, title, description, type,
|
|
38
|
+
enum, const, anyOf, properties, required, additionalProperties, items, minItems,
|
|
39
|
+
maxItems, minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf,
|
|
40
|
+
minLength, maxLength and pattern. The root must declare type object. Nested nullable
|
|
41
|
+
values and local JSON Pointer references are supported; external/cyclic references,
|
|
42
|
+
unknown keywords, invalid patterns and invalid schemas are rejected before model
|
|
43
|
+
or Tool execution. The dialect is JSON Schema 2020-12. Schema input accepts JSON
|
|
44
|
+
values and String/Symbol keys, snapshots recursively, and rejects duplicate normalized
|
|
45
|
+
keys, Ruby objects and cycles. Generic schema interpretation belongs to C17, not
|
|
46
|
+
Tool::Schema, Agent's caller or Generation.
|
|
47
|
+
|
|
48
|
+
## Native transport and completion facts
|
|
49
|
+
|
|
50
|
+
C09 `LLMAdapter::Request#response_schema` carries the immutable schema; each model
|
|
51
|
+
turn retains it. RubyLLM `with_schema` sends native schema configuration. The
|
|
52
|
+
transport tests cover RubyLLM 2.0.0 OpenAI/Azure Chat Completions and Responses,
|
|
53
|
+
Anthropic, Gemini and Ollama's OpenAI-compatible endpoint. Constraints are compared
|
|
54
|
+
with the actual HTTP payload. SDK removal of root schema/title annotations does not
|
|
55
|
+
change validation. These are offline protocol tests, not certifications of every
|
|
56
|
+
remote model. Known model capability failures and native schema rejection raise
|
|
57
|
+
`LLMAdapter::StructuredOutputUnsupportedError` (`llm.structured_output_unsupported`).
|
|
58
|
+
Unregistered OpenAI-compatible local models may attempt native schema delivery.
|
|
59
|
+
Other providers are rejected explicitly for structured requests until their native
|
|
60
|
+
transport has been verified. There is no downgrade to JSON mode or prompt-only
|
|
61
|
+
instructions. Tool capability is checked too when the model catalog is available;
|
|
62
|
+
remote schema/Tool combination rejection remains a provider failure.
|
|
63
|
+
|
|
64
|
+
LM Studio's HTTP 400 string error `Cannot combine structured output constraints
|
|
65
|
+
with lazy grammar` is a known native schema/Tool rejection. It is classified as
|
|
66
|
+
`StructuredOutputUnsupportedError` with its original cause, for both complete and
|
|
67
|
+
stream operations. Unrelated string errors and requests without a schema retain
|
|
68
|
+
their ordinary transport-error behavior. The adapter neither downgrades schemas
|
|
69
|
+
nor replays requests. RubyLLM's configured transport retry policy is independent.
|
|
70
|
+
The minimum Faraday streaming callback omits the HTTP status, so RubyLLM may
|
|
71
|
+
surface this same exact rejection as `ServerError`; its cause is preserved too.
|
|
72
|
+
That synthetic server status can trigger SDK retries before the adapter receives
|
|
73
|
+
the error. Adapter replay tests disable SDK retries to verify their own boundary.
|
|
74
|
+
|
|
75
|
+
`Response` records finish_reason (stop, tool_calls, length, content_filter, refusal,
|
|
76
|
+
unknown), refusal and raw_finish_reason metadata. Only normal final completion can
|
|
77
|
+
be a structured success. Tool requests are intermediate; they retain existing
|
|
78
|
+
approval, denial, cancellation and recovery behavior. RubyLLM 2.0 loses streamed
|
|
79
|
+
refusal fields in Chunk values; operation-local Protocol subclasses capture these
|
|
80
|
+
raw facts. No SDK global class or shared protocol registry is patched. The SDK's
|
|
81
|
+
raw finish information is retained through saved-response recovery.
|
|
82
|
+
|
|
83
|
+
JSON 2.21.2 is the verified minimum for the strict decoding flags and deterministic
|
|
84
|
+
schema serialization; the minimum dependency CI pins JSON 2.21.2 and RubyLLM 2.0.0
|
|
85
|
+
alongside the existing minimum HTTP dependencies. Unknown SDK or parser faults
|
|
86
|
+
remain failures, not poor-quality Generation results.
|
|
87
|
+
|
|
88
|
+
## Failure and persistence
|
|
89
|
+
|
|
90
|
+
`Agent::OutputError < Phronomy::Error` exposes execution_id, schema_digest, reason,
|
|
91
|
+
violations and raw_output. Violations contain pointers/keywords without embedding
|
|
92
|
+
model text. Raw text is explicitly available for controlled diagnostics; generic
|
|
93
|
+
error messages do not interpolate it.
|
|
94
|
+
|
|
95
|
+
| Code | Reasons |
|
|
96
|
+
| --- | --- |
|
|
97
|
+
| agent.output_invalid | invalid_json, schema_mismatch, missing_output |
|
|
98
|
+
| agent.output_refused | model_refusal |
|
|
99
|
+
| agent.output_incomplete | length, content_filter, unknown_finish_reason |
|
|
100
|
+
|
|
101
|
+
Agent owns error translation. Parser invalid_json/schema_mismatch become OutputError
|
|
102
|
+
with the original cause. A saved error is restored by its explicit owner codec;
|
|
103
|
+
no arbitrary class name is constantized. Other transport, cancellation, Tool and
|
|
104
|
+
internal failures keep their existing behavior. Output failure never triggers an
|
|
105
|
+
automatic regeneration or replay of a completed Tool operation.
|
|
106
|
+
|
|
107
|
+
Execution metadata stores `output_contract` version 1, schema_ref and schema_digest
|
|
108
|
+
at admission. Digest is SHA-256 of UTF-8 JSON generated with recursively sorted keys
|
|
109
|
+
(JSON 2.21.2 `sort_keys: true`); the exact same representation is saved in ContentStore.
|
|
110
|
+
This format preserves arbitrary JSON integers and finite floats, including -0.0,
|
|
111
|
+
without applying the narrower Common canonical-domain number rules. Each Context
|
|
112
|
+
Manifest retains response_schema_ref, including budget projection and recovery.
|
|
113
|
+
New executions may choose another schema; existing executions cannot replace their
|
|
114
|
+
schema on resume/approval or by reusing a reserved ID with a different digest.
|
|
115
|
+
Generic config/provider schema overrides are rejected.
|
|
116
|
+
|
|
117
|
+
Structured success stores JSON result plus raw_output_ref and result_format=json.
|
|
118
|
+
Old records without output_contract remain facts from a free-text execution; no
|
|
119
|
+
schema is added retroactively. The execution codec stays at 0.2: its extensible
|
|
120
|
+
metadata contains the explicitly versioned output contract. Unknown output-contract
|
|
121
|
+
versions or digest mismatch fail closed. Saved LLM responses include finish_reason,
|
|
122
|
+
refusal and metadata; missing finish evidence is unknown, never assumed normal.
|
|
123
|
+
Fresh-runtime and new SQL connection-pool tests verify success, failure and approval
|
|
124
|
+
continuation without replay of confirmed LLM/Tool work. This is a durable-facts
|
|
125
|
+
claim, not an external side-effect exactly-once guarantee.
|
|
126
|
+
|
|
127
|
+
## Generation evaluation
|
|
128
|
+
|
|
129
|
+
C03 specifies the fixed protocol: draft answer/string, confidence/number in 0..1,
|
|
130
|
+
citations/array; review approved/boolean, score/number in 0..1, feedback/string.
|
|
131
|
+
All outer fields are required and additional properties are forbidden. Default
|
|
132
|
+
citation objects require source and excerpt strings. `citation_schema:` customizes
|
|
133
|
+
the citation object only; local references are rebased when embedded, without
|
|
134
|
+
rewriting const/enum data. Generic validation remains with Agent/C17.
|
|
135
|
+
|
|
136
|
+
The old draft_parser/review_parser arguments, DefaultParser, ParseError fallback,
|
|
137
|
+
score clamping and Generation parser composition wiring are removed. C03 receives
|
|
138
|
+
validated String-key values and performs domain evaluation only. Invalid output
|
|
139
|
+
returns `Result` with termination_reason=output_failed, interrupted_phase, failure,
|
|
140
|
+
confidence=nil and review_approved=nil. Failed draft clears any previous answer;
|
|
141
|
+
failed review retains the current valid draft. iterations counts successful drafts.
|
|
142
|
+
A valid score of zero can still satisfy a zero threshold; invalid data cannot.
|
|
143
|
+
Normal low-quality output follows the configured evaluation loop as before.
|
|
144
|
+
|
|
145
|
+
`Result#failure` is nil except for output_failed, where it carries the owned error
|
|
146
|
+
diagnostic (without raw_output). Hash/JSON round trips, equality, hashing and freeze
|
|
147
|
+
include it. `Result.from_h` requires every new field; migrate old saved Result JSON
|
|
148
|
+
explicitly with `scripts/migrate_generator_verifier_result.rb OLD.json NEW.json`.
|
|
149
|
+
The script adds failure=nil, validates the result and creates a new file. It does
|
|
150
|
+
not overwrite the source or silently alter invalid data. `raise_if_unaccepted`
|
|
151
|
+
retains the Result and the original OutputError cause in the same process.
|
|
152
|
+
|
|
153
|
+
## Symmetric lifetime rules
|
|
154
|
+
|
|
155
|
+
`Agent.open_scope` and `GeneratorVerifier.open` wait exactly once when the direct
|
|
156
|
+
block return is a TaskResult. They then attempt all releases. Other values, arrays
|
|
157
|
+
and nested Tasks are not recursively awaited. Failure/cancellation is re-raised
|
|
158
|
+
with the same identity if cleanup succeeds. Double failure retains primary_error,
|
|
159
|
+
all cleanup failures, the scope/pipeline recovery handle and the primary cause.
|
|
160
|
+
`result` contains only a resolved successful value, never a failed Task.
|
|
161
|
+
|
|
162
|
+
`Scope#release!` and GeneratorVerifier release retain failed owners for retry and
|
|
163
|
+
remove successful ones. Return true on complete release, false after release.
|
|
164
|
+
Partial release rejects new work/acquisition. Concurrent Scope release raises
|
|
165
|
+
agent.scope_busy; interruption of a releasing call leaves a retryable failed state.
|
|
166
|
+
Materialization receipts are drained to physical completion before release.
|
|
167
|
+
Neither API implicitly cancels work, answers approval, deletes saved data or
|
|
168
|
+
closes the application Store. The application decides how unfinished work becomes
|
|
169
|
+
safe before retrying through the cleanup error's scope/pipeline handle.
|
|
170
|
+
|
|
171
|
+
## Dependencies and implementation units
|
|
172
|
+
|
|
173
|
+
- Remove C03 Generation → C17 OutputParser; C03 continues to use public C04 Agent,
|
|
174
|
+
C02 Workflow and C13 Execution contracts.
|
|
175
|
+
- Add C04 Agent → C17 OutputParser as a default implementation composed at boot.
|
|
176
|
+
C04 owns the output promise and error; C17 owns syntax/schema interpretation.
|
|
177
|
+
- C09 transports schema/final-response facts and remains independent of C17/C03.
|
|
178
|
+
- No new Storage, Engine or Runtime-private dependency from Generation is added.
|
|
179
|
+
|
|
180
|
+
The Contract concept/foundation SVGs reflect these two edges. Generation → Agent
|
|
181
|
+
must retain its existing route above OutputParser. Repository source diagrams are
|
|
182
|
+
regenerated by the Architecture tool; conceptual target diagrams remain distinct
|
|
183
|
+
from the complete source graph.
|
|
@@ -162,9 +162,9 @@ This guarantee is specific to Framework automatic instrumentation. It does not
|
|
|
162
162
|
redefine the behavior of Application code that directly calls the public
|
|
163
163
|
`Tracing::Base#trace` template method.
|
|
164
164
|
|
|
165
|
-
## 10.
|
|
165
|
+
## 10. Observed token counts
|
|
166
166
|
|
|
167
|
-
`
|
|
167
|
+
Tracing receives a plain `Hash[Symbol, Integer?]` in `usage:`. Supported keys are:
|
|
168
168
|
|
|
169
169
|
```text
|
|
170
170
|
input
|
|
@@ -173,8 +173,26 @@ cached
|
|
|
173
173
|
cache_creation
|
|
174
174
|
```
|
|
175
175
|
|
|
176
|
-
A backend may export the subset its telemetry schema supports.
|
|
177
|
-
|
|
176
|
+
A backend may export the subset its telemetry schema supports. Missing/unknown
|
|
177
|
+
counts remain nil and zero remains a measurement. Tracing does not accept or
|
|
178
|
+
interpret a domain-specific usage object. The LLM domain retains ownership of
|
|
179
|
+
its immutable usage value and aggregation rules; its caller projects the value
|
|
180
|
+
with `to_h` before sending it to Tracing. Agent owns that projection for both
|
|
181
|
+
`llm.call` and `agent.execution`. Arbitrary Workflow/Tool results are not inspected
|
|
182
|
+
for a `usage` field by Tracing.
|
|
183
|
+
|
|
184
|
+
For manual tracing, change `[result, usage]` to `[result, usage&.to_h]` when the
|
|
185
|
+
second value is a domain object. Custom exporters read `usage[:input]` and
|
|
186
|
+
`usage[:output]` instead of calling methods on the domain object. No compatibility
|
|
187
|
+
conversion inside Tracing is retained: it would preserve the dependency being
|
|
188
|
+
removed. The `usage:` keyword, span lifecycle and external telemetry field names
|
|
189
|
+
are unchanged.
|
|
190
|
+
|
|
191
|
+
The internal automatic task observer accepts an owner-supplied projection. A
|
|
192
|
+
failed projection is diagnostic-only: it records no usage and does not change
|
|
193
|
+
the operation result or prevent the span from finishing. Tracing has no static
|
|
194
|
+
dependency on LLMAdapter. This does not assign LLM retry policy to Tracing or to
|
|
195
|
+
the common LLM contract.
|
|
178
196
|
|
|
179
197
|
## 11. Backend notes
|
|
180
198
|
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# RubyLLM transport error boundaries
|
|
2
|
+
|
|
3
|
+
## Purpose and ownership
|
|
4
|
+
|
|
5
|
+
Applications need to handle a failed provider communication through Phronomy's
|
|
6
|
+
public API without knowing the HTTP client hidden behind the selected backend.
|
|
7
|
+
LLM generation and embedding are separate contracts, so each retains its own
|
|
8
|
+
existing `TransportError`. The concrete RubyLLM backends translate identifiable
|
|
9
|
+
communication failures; Agent, Generation and Execution propagate that result.
|
|
10
|
+
|
|
11
|
+
This change adds no public class, parameter, failure code, cross-contract edge,
|
|
12
|
+
shared error translator or retry mechanism. Faraday references remain in the two
|
|
13
|
+
concrete backends. The public RBS shape is unchanged.
|
|
14
|
+
|
|
15
|
+
## Mapping and exclusions
|
|
16
|
+
|
|
17
|
+
| Failure at the RubyLLM boundary | LLM complete/stream | Embeddings |
|
|
18
|
+
| --- | --- | --- |
|
|
19
|
+
| `Faraday::TimeoutError` | `LLMAdapter::TransportError` | `Embeddings::TransportError` |
|
|
20
|
+
| `Faraday::ConnectionFailed` | `LLMAdapter::TransportError` | `Embeddings::TransportError` |
|
|
21
|
+
| Existing RubyLLM errors | Existing authentication/rate-limit/context/transport classifications | Existing authentication/rate-limit/transport classifications |
|
|
22
|
+
| Application streaming callback, including either Faraday class above | Original exception identity | Not applicable |
|
|
23
|
+
| Cancellation, execution deadline, arbitrary I/O, programming failure or unrecognized Faraday error | Original exception identity | Original exception identity |
|
|
24
|
+
|
|
25
|
+
The top-level mapped exception preserves the original message and its SDK/HTTP
|
|
26
|
+
exception as `cause`. That exception's network cause is retained, for example:
|
|
27
|
+
`LLMAdapter::TransportError -> Faraday::TimeoutError -> Net::ReadTimeout`.
|
|
28
|
+
No classification is inferred from an exception's message or nested cause.
|
|
29
|
+
The explicitly selected LM Studio backend additionally decodes the HTTP response
|
|
30
|
+
body at the SDK boundary using a narrow provider-response mapping; see
|
|
31
|
+
[LM Studio output failures](lm-studio-output-failures.md). This is not a generic
|
|
32
|
+
exception-message or cause-chain heuristic, and the default RubyLLM mapping below
|
|
33
|
+
is unchanged.
|
|
34
|
+
Generic `Faraday::Error`, `Faraday::SSLError`, raw `IOError` and system errors are
|
|
35
|
+
not added to this mapping. Future additions require evidence and an explicit
|
|
36
|
+
category decision rather than a blanket rescue.
|
|
37
|
+
|
|
38
|
+
The LLM backend's callback identity check runs before provider classification.
|
|
39
|
+
An application callback throwing `Faraday::TimeoutError` therefore remains the
|
|
40
|
+
same object. A provider timeout after a delivered chunk is a transport failure;
|
|
41
|
+
the already delivered chunk is not withdrawn, replayed or turned into success.
|
|
42
|
+
|
|
43
|
+
## Application code before and after
|
|
44
|
+
|
|
45
|
+
Before, handling a provider timeout required knowledge of the backend's client:
|
|
46
|
+
|
|
47
|
+
```ruby
|
|
48
|
+
begin
|
|
49
|
+
response = adapter.complete(request)
|
|
50
|
+
rescue Phronomy::LLMAdapter::TransportError,
|
|
51
|
+
Faraday::TimeoutError, Faraday::ConnectionFailed => error
|
|
52
|
+
warn "Provider communication failed: #{error.message}"
|
|
53
|
+
end
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
After, the same communication categories are handled through the public owner:
|
|
57
|
+
|
|
58
|
+
```ruby
|
|
59
|
+
begin
|
|
60
|
+
response = adapter.complete(request)
|
|
61
|
+
rescue Phronomy::LLMAdapter::TransportError => error
|
|
62
|
+
warn "Provider communication failed: #{error.message}"
|
|
63
|
+
end
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Embeddings applications use `Phronomy::Embeddings::TransportError` in the same
|
|
67
|
+
way. Applications may inspect `cause` for diagnostics, but ordinary transport
|
|
68
|
+
handling does not need a Faraday dependency. Existing specialized errors remain
|
|
69
|
+
available for applications that distinguish authentication or rate limiting.
|
|
70
|
+
|
|
71
|
+
## Retry, interruption and lifetime
|
|
72
|
+
|
|
73
|
+
The accepted [transport-policy ADR](../decisions/011-delegate-transport-policy-to-adapters.md)
|
|
74
|
+
continues to apply. RubyLLM owns its configured HTTP retries. Phronomy translates
|
|
75
|
+
the failure reaching its boundary and does not replay the LLM operation, Agent
|
|
76
|
+
invocation or Tool work. A transport classification does not assert that no
|
|
77
|
+
output or external effect occurred, and does not authorize a retry.
|
|
78
|
+
|
|
79
|
+
An execution deadline and an HTTP timeout remain different failures. This change
|
|
80
|
+
does not force worker threads to stop, alter cancellation, or make an active
|
|
81
|
+
Agent safe to unload. Once an ordinary provider failure has settled and its
|
|
82
|
+
physical work has finished, the existing lifetime APIs can release its owners.
|
|
83
|
+
|
|
84
|
+
The old Faraday 2.1.0 / faraday-net_http 2.1.0 streaming path lost HTTP status and
|
|
85
|
+
caused RubyLLM to retry a schema/Tool rejection as a synthetic ServerError.
|
|
86
|
+
The supported floors are now Faraday 2.14.4 / faraday-net_http 3.4.4. Real HTTP
|
|
87
|
+
tests verify the original status and request count with SDK retries enabled.
|
|
88
|
+
The obsolete ServerError-to-schema-rejection fallback is removed: an actual 500
|
|
89
|
+
remains a server failure even when its text resembles a known 400 rejection.
|
|
90
|
+
Configured SDK retries for 429/500 remain intact; no global retry switch or
|
|
91
|
+
SDK monkey patch is introduced.
|
|
92
|
+
|
|
93
|
+
## Verification
|
|
94
|
+
|
|
95
|
+
Unit contracts cover both mapped types across complete, stream and embed,
|
|
96
|
+
identity exclusions, same-class callback failures and failure after partial
|
|
97
|
+
output. Loopback HTTP tests exercise the real RubyLLM/Faraday stack for timeout,
|
|
98
|
+
connection loss, partial streaming with SDK retries enabled, callback identity,
|
|
99
|
+
and GeneratorVerifier propagation/release. The minimum HTTP CI job runs these
|
|
100
|
+
contracts too. Test-only shortened timeouts do not change production settings.
|
|
101
|
+
|
|
102
|
+
These tests do not certify live model answer quality, schema/Tool combination
|
|
103
|
+
support, or the root cause of the earlier unreproduced LM Studio wait.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# Trust pipeline acceptance follow-up
|
|
2
|
+
|
|
3
|
+
## Purpose and responsibility
|
|
4
|
+
|
|
5
|
+
Communication integrity, schema validity and business truth are different
|
|
6
|
+
promises. Successful HTTP and valid JSON cannot establish faithful extraction.
|
|
7
|
+
|
|
8
|
+
| Concern | Owner | Change |
|
|
9
|
+
| --- | --- | --- |
|
|
10
|
+
| HTTP status and configured retries | HTTP stack / concrete RubyLLM backend | Require a verified stack that preserves streaming status |
|
|
11
|
+
| Schema-valid terminal values | Agent, using OutputParser | Existing guarantee retained |
|
|
12
|
+
| Evaluation criteria and result meaning | GeneratorVerifier | Existing protocol retained |
|
|
13
|
+
| Exact invoice extraction | Application / acceptance oracle | Reject incorrect values explicitly |
|
|
14
|
+
| Approval decision and Tool effects | Application decision, Agent execution, Generation routing | Observe notification, decision, effects and release |
|
|
15
|
+
| Unexplained wait | Diagnostic runner plus server logs | Record payload, settings, attempts and periodic thread stacks |
|
|
16
|
+
|
|
17
|
+
Contract owners, their dependencies, Engine and Execution are unchanged.
|
|
18
|
+
No SDK monkey patch, extra retry loop, output repair or schema downgrade is added.
|
|
19
|
+
|
|
20
|
+
## HTTP dependency correction
|
|
21
|
+
|
|
22
|
+
Faraday 2.1.0 with faraday-net_http 2.1.0 did not pass the response environment to
|
|
23
|
+
streaming callbacks. RubyLLM 2.0.0 could interpret a JSON 400 as a synthetic 500.
|
|
24
|
+
401 and 429 could also lose their identity. The ServerError fallback additionally
|
|
25
|
+
misclassified an actual 500 whose text matched the known schema rejection.
|
|
26
|
+
|
|
27
|
+
Supported floors are now Faraday 2.14.4 and faraday-net_http 3.4.4, the pair
|
|
28
|
+
already verified on the normal path. Both are explicit in the gemspec: raising
|
|
29
|
+
Faraday alone does not exclude older Net::HTTP adapters. CI pins and checks both,
|
|
30
|
+
with the existing minimum RubyLLM, MCP and JSON versions. These are verified
|
|
31
|
+
support floors, not claims about the earliest upstream fix. Custom Faraday
|
|
32
|
+
adapters need their own streaming/status compatibility assessment.
|
|
33
|
+
|
|
34
|
+
The obsolete ServerError fallback is removed. Only a real BadRequestError with
|
|
35
|
+
the known body becomes StructuredOutputUnsupportedError. Actual 500 responses
|
|
36
|
+
retain the SDK server-error cause. Configured SDK retries for 429/500 remain;
|
|
37
|
+
a 400 rejection is one HTTP attempt even with three retries configured.
|
|
38
|
+
Partial-stream replay and callback identity regressions remain in place.
|
|
39
|
+
|
|
40
|
+
Before and after, application handling uses the same public API:
|
|
41
|
+
|
|
42
|
+
```ruby
|
|
43
|
+
rescue Phronomy::LLMAdapter::StructuredOutputUnsupportedError => error
|
|
44
|
+
warn error.cause.class
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Before, the old stack could issue four requests and expose a synthetic ServerError
|
|
48
|
+
for HTTP 400. After, the cause is BadRequestError and that rejection is not
|
|
49
|
+
retried as a transient server failure.
|
|
50
|
+
|
|
51
|
+
## Content fidelity and acceptance
|
|
52
|
+
|
|
53
|
+
The invoice demo keeps its general string/number schema and asks the model to
|
|
54
|
+
preserve identifier characters. After Agent release, an independent oracle checks
|
|
55
|
+
the fixed fixture. Incorrect values exit 1; they are never replaced or retried.
|
|
56
|
+
This is fixture acceptance, not a general invoice validator or Agent responsibility.
|
|
57
|
+
|
|
58
|
+
Before, INV-2026-01 could print and exit successfully for source INV-2026-001.
|
|
59
|
+
After, it is a fidelity failure, while schema validation remains correctly
|
|
60
|
+
successful. One correct response does not establish general model accuracy.
|
|
61
|
+
|
|
62
|
+
The examples runner checks invoice, accepted, iteration_limit and approval
|
|
63
|
+
yes/no/cancel independently. Approval requires a notification and the expected
|
|
64
|
+
Tool effects. A schema/Tool rejection is blocked, not successful approval.
|
|
65
|
+
Yes checks approval and valid completion; accepted is a separate test because
|
|
66
|
+
Tool approval does not guarantee that evaluation criteria will be satisfied.
|
|
67
|
+
|
|
68
|
+
## Wait diagnosis and limits
|
|
69
|
+
|
|
70
|
+
The opt-in runner observes the pinned SDK without replacing the provider or
|
|
71
|
+
changing timeout/retry settings. It records UTC times, request payloads/hashes,
|
|
72
|
+
settings, physical HTTP attempts, results, exception causes, releases and periodic
|
|
73
|
+
Ruby thread stacks. Observer failures invalidate the report and do not replace
|
|
74
|
+
the application exception. Signals wrapped by cleanup errors remain interrupted.
|
|
75
|
+
|
|
76
|
+
Observation uses diagnostic-only SDK method bindings and must be reviewed when
|
|
77
|
+
the SDK changes. Reports contain prompts/responses. Sampling and file writes add
|
|
78
|
+
overhead; this runner is not a performance benchmark.
|
|
79
|
+
|
|
80
|
+
Historical LM Studio logs cannot be reconstructed. Match the saved review payload
|
|
81
|
+
and UTC times with server logs; use future wait stacks to identify pending
|
|
82
|
+
HTTP/SDK/framework operations before assigning a cause. A later fast response
|
|
83
|
+
does not resolve the earlier wait.
|
|
84
|
+
|
|
85
|
+
Controlled HTTP fixtures prove framework and observation behavior, not live
|
|
86
|
+
model accuracy or server support for schema plus Tools. Live acceptance still
|
|
87
|
+
requires a suitable model/server environment and recorded server version.
|
|
@@ -153,7 +153,7 @@ request deduplication or unambiguous correlation. Existing retention applies.
|
|
|
153
153
|
|
|
154
154
|
Reuse existing APIs where they meet this contract; map missing capabilities only
|
|
155
155
|
after baseline inspection. Do not invent a framework outbox or request registry.
|
|
156
|
-
See [RC-01](RECOVERY_CONTRACT_CLARIFICATIONS.md#1-rc-01
|
|
156
|
+
See [RC-01](../design/durable-semantic-coordination/RECOVERY_CONTRACT_CLARIFICATIONS.md#1-rc-01-accessing-final-results-and-discovering-executions).
|
|
157
157
|
|
|
158
158
|
### 9. Unknown commit outcome and cancellation
|
|
159
159
|
|
|
@@ -1,33 +1,33 @@
|
|
|
1
|
-
# V2 revision 2
|
|
1
|
+
# V2 revision 2 - Changes
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Updated: 2026-09-06 / Status: Accepted
|
|
4
4
|
|
|
5
|
-
##
|
|
5
|
+
## Accepted decisions reflected in this revision
|
|
6
6
|
|
|
7
|
-
|
|
|
7
|
+
| Item | Change in this revision |
|
|
8
8
|
|---|---|
|
|
9
|
-
| RC-01
|
|
10
|
-
| RC-02
|
|
11
|
-
| RC-03
|
|
12
|
-
| RC-04
|
|
13
|
-
| RC-05
|
|
9
|
+
| RC-01 Result access | Add contracts for reading status/results by exact execution ID and discovering candidates from the owner if execution is interrupted before the caller receives the ID |
|
|
10
|
+
| RC-02 Unknown persistence outcomes | Distinguish confirmed absence, read failure, CAS conflict, and unknown commit outcome; reconcile using the same reserved identity |
|
|
11
|
+
| RC-03 Wiring compatibility | Specify minimum checks for definition id/version, slot, graph, and Persistence boundary, and give precedence to committed facts |
|
|
12
|
+
| RC-04 Cancellation | Distinguish ending observation/shutdown from semantic cancellation; specify handling by child state, races, and rediscovery after restart |
|
|
13
|
+
| RC-05 Guarantee wording | Use consistent wording: reuse committed outcomes, recover with the same identity, and resolve uncertain external outcomes through existing Recovery |
|
|
14
14
|
|
|
15
|
-
ADR-029/030/031
|
|
16
|
-
|
|
17
|
-
|
|
15
|
+
The corresponding normative statements have been incorporated into ADR-029/030/031, the implementation design, the responsibility boundary review, and the README.
|
|
16
|
+
Detailed contracts and validation conditions are collected in RECOVERY_CONTRACT_CLARIFICATIONS.md and referenced across the documents.
|
|
17
|
+
The pre-approval Proposed status has been changed to Accepted. This does not request renewed approval of the new functionality.
|
|
18
18
|
|
|
19
|
-
##
|
|
19
|
+
## Correction of implementation status
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
21
|
+
Earlier implementation counts and successful static checks in the handoff document remain reports from the previous session.
|
|
22
|
+
They are not evidence that implementation was revalidated in this continuation. Only the design documents and handoff document are currently available;
|
|
23
|
+
the old overlay/apply.py/APPLY.sh/VERIFY.sh have not been recovered. The resumption procedure has also been updated to begin by recovering the actual files.
|
|
24
24
|
|
|
25
|
-
##
|
|
25
|
+
## Checks performed for this revision
|
|
26
26
|
|
|
27
|
-
-
|
|
28
|
-
-
|
|
29
|
-
- RC-01
|
|
30
|
-
- ZIP
|
|
27
|
+
- Check consistency of approval status, baseline, and cross-references in the affected documents.
|
|
28
|
+
- Correct obsolete statements that implementation was stopped pending approval and guarantee wording that could imply exactly-once external effects.
|
|
29
|
+
- Check the mapping of RC-01 through RC-05 to ADRs, implementation design, and validation conditions.
|
|
30
|
+
- Verify the ZIP layout, SHA256SUMS for all documents, and external SHA256 values for the ZIP and handoff document.
|
|
31
31
|
|
|
32
|
-
|
|
33
|
-
API
|
|
32
|
+
No code changes or RSpec runs were performed in the target repository.
|
|
33
|
+
Unverified API or exception names were not invented; mapping the requirements to the baseline was made mandatory before implementation.
|