riffer 0.35.0 → 0.36.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/CHANGELOG.md +11 -0
- data/README.md +0 -1
- data/docs/01_OVERVIEW.md +0 -1
- data/docs/03_AGENTS.md +3 -7
- data/docs/10_CONFIGURATION.md +0 -17
- data/lib/riffer/agent/run.rb +11 -35
- data/lib/riffer/agent.rb +1 -1
- data/lib/riffer/config.rb +0 -45
- data/lib/riffer/guardrails/runner.rb +7 -32
- data/lib/riffer/providers/anthropic.rb +1 -1
- data/lib/riffer/providers/base.rb +11 -75
- data/lib/riffer/tools/runtime.rb +8 -29
- data/lib/riffer/version.rb +1 -1
- data/sig/_private/opentelemetry.rbs +3 -6
- data/sig/generated/riffer/agent/run.rbs +4 -8
- data/sig/generated/riffer/agent.rbs +1 -1
- data/sig/generated/riffer/config.rbs +0 -33
- data/sig/generated/riffer/guardrails/runner.rbs +0 -8
- data/sig/generated/riffer/providers/base.rbs +2 -20
- data/sig/generated/riffer/tools/runtime.rbs +2 -6
- metadata +5 -16
- data/docs/17_METRICS.md +0 -155
- data/lib/riffer/metrics/instruments.rb +0 -31
- data/lib/riffer/metrics/no_op.rb +0 -14
- data/lib/riffer/metrics/otel.rb +0 -80
- data/lib/riffer/metrics.rb +0 -94
- data/sig/generated/riffer/metrics/instruments.rbs +0 -15
- data/sig/generated/riffer/metrics/no_op.rbs +0 -10
- data/sig/generated/riffer/metrics/otel.rbs +0 -48
- data/sig/generated/riffer/metrics.rbs +0 -72
- data/sig/manual/riffer/metrics/no_op.rbs +0 -5
- data/sig/manual/riffer/metrics.rbs +0 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 491260200757bf1d17b920f099012d771328fd333d46bb325f53dca863bcf204
|
|
4
|
+
data.tar.gz: 072ffcdf47b2aa115a3c7743cb77d4f238d18b926f0a53eb6b45a28148b8b2be
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 97bf5aeed05fe9b591ae5a2bcf7c59d6990a46de1695fa4ff929823ead08b23372f35c4efb389a7e2f769a2a15dbad4dcb35bb1421e57c1fe5291216390b176c
|
|
7
|
+
data.tar.gz: 421d381a7d8c0a1be0c017a2c347096fa452d48d512706c474e82b5d90097e245d9f7680ed6745c10aac31502cdfe47efc56ae885dbd9e4b52b3b7e2803b7c10
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,17 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.36.0](https://github.com/janeapp/riffer/compare/riffer/v0.35.0...riffer/v0.36.0) (2026-07-03)
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
### ⚠ BREAKING CHANGES
|
|
12
|
+
|
|
13
|
+
* config.metrics, the Riffer::Metrics module, and Riffer::Metrics::Otel are removed. Hosts wiring a metrics backend must drop that configuration.
|
|
14
|
+
|
|
15
|
+
### Features
|
|
16
|
+
|
|
17
|
+
* remove the metrics primitive ([#347](https://github.com/janeapp/riffer/issues/347)) ([70a838e](https://github.com/janeapp/riffer/commit/70a838e9d49da12f89886c6e6f7bca35f427f77c))
|
|
18
|
+
|
|
8
19
|
## [0.35.0](https://github.com/janeapp/riffer/compare/riffer/v0.34.0...riffer/v0.35.0) (2026-06-25)
|
|
9
20
|
|
|
10
21
|
|
data/README.md
CHANGED
|
@@ -63,7 +63,6 @@ For comprehensive documentation, see the [docs](docs/) directory:
|
|
|
63
63
|
- [MCP](docs/14_MCP.md) - Integrating third-party MCP servers
|
|
64
64
|
- [Serialization](docs/15_SERIALIZATION.md) - Persisting and transferring agent definitions
|
|
65
65
|
- [Tracing](docs/16_TRACING.md) - OpenTelemetry span contract and host wiring
|
|
66
|
-
- [Metrics](docs/17_METRICS.md) - OpenTelemetry metric contract and host wiring
|
|
67
66
|
- [Providers](docs/providers/01_PROVIDERS.md) - LLM provider adapters
|
|
68
67
|
|
|
69
68
|
### API Reference
|
data/docs/01_OVERVIEW.md
CHANGED
|
@@ -132,7 +132,6 @@ Response
|
|
|
132
132
|
- [Tools](06_TOOLS.md) - Creating tools
|
|
133
133
|
- [Configuration](10_CONFIGURATION.md) - Global configuration
|
|
134
134
|
- [Tracing](16_TRACING.md) - OpenTelemetry span contract and host wiring
|
|
135
|
-
- [Metrics](17_METRICS.md) - OpenTelemetry metric contract and host wiring
|
|
136
135
|
- [Evals](11_EVALS.md) - Evaluating agent quality
|
|
137
136
|
- [Guardrails](12_GUARDRAILS.md) - Input/output validation
|
|
138
137
|
- [Skills](13_SKILLS.md) - Packaged agent capabilities
|
data/docs/03_AGENTS.md
CHANGED
|
@@ -333,7 +333,7 @@ Keys and values may be `String` or `Symbol`; both are stringified, and entries w
|
|
|
333
333
|
Tags propagate to **two** places:
|
|
334
334
|
|
|
335
335
|
1. The provider's native per-request metadata field (see the mapping below).
|
|
336
|
-
2. Observability — stamped as `riffer.tag.<key>` on **every** span
|
|
336
|
+
2. Observability — stamped as `riffer.tag.<key>` on **every** span the call emits (`invoke_agent`, `chat`, `execute_tool`, `execute_guardrail`). See [Tracing](16_TRACING.md).
|
|
337
337
|
|
|
338
338
|
### Reserved key: `user_id`
|
|
339
339
|
|
|
@@ -349,18 +349,14 @@ Tags propagate to **two** places:
|
|
|
349
349
|
| Anthropic | `metadata.user_id` **only** | The only tag forwarded |
|
|
350
350
|
| Gemini | _(none — observability only)_ | Tag only; no request field |
|
|
351
351
|
|
|
352
|
-
**Anthropic silently drops non-`user_id` tags.** The Messages API has no free-form request-metadata field — only `metadata.user_id`. So for Anthropic, `user_id` is forwarded as `metadata: {user_id: …}` and **every other tag is dropped from the request** (it still appears on spans
|
|
352
|
+
**Anthropic silently drops non-`user_id` tags.** The Messages API has no free-form request-metadata field — only `metadata.user_id`. So for Anthropic, `user_id` is forwarded as `metadata: {user_id: …}` and **every other tag is dropped from the request** (it still appears on spans). This is intentional.
|
|
353
353
|
|
|
354
|
-
**Gemini is observability-only.** Riffer's Gemini adapter targets the Gemini Developer API (`generativelanguage.googleapis.com`), whose `generateContent` request has **no** `labels` field — sending unknown fields is rejected. So tags are **not** added to the Gemini request; they propagate to spans
|
|
354
|
+
**Gemini is observability-only.** Riffer's Gemini adapter targets the Gemini Developer API (`generativelanguage.googleapis.com`), whose `generateContent` request has **no** `labels` field — sending unknown fields is rejected. So tags are **not** added to the Gemini request; they propagate to spans only. Native request labels (`labels`, lowercase `[a-z0-9_-]`, ≤63 chars each) are a Vertex AI feature and would arrive with a future Vertex adapter.
|
|
355
355
|
|
|
356
356
|
### Provider limits (not enforced)
|
|
357
357
|
|
|
358
358
|
Riffer does not validate tag count, key/value length, or charset — it forwards what you give it and lets the provider reject anything out of bounds. Known limits: Bedrock `requestMetadata` ≤16 entries, key/value ≤256 chars; OpenAI / OpenRouter `metadata` ≤16 pairs (OpenRouter: 64-char keys, 512-char values).
|
|
359
359
|
|
|
360
|
-
### Metric cardinality
|
|
361
|
-
|
|
362
|
-
Tags are promoted onto metric dimensions verbatim — there is no allow-list. High-cardinality values (e.g. a unique `user_id` per request) will create a correspondingly large number of metric series in your backend. Keep metric-bound tags low-cardinality.
|
|
363
|
-
|
|
364
360
|
## Expand Your Agent
|
|
365
361
|
|
|
366
362
|
| Goal | Feature | Guide |
|
data/docs/10_CONFIGURATION.md
CHANGED
|
@@ -119,23 +119,6 @@ end
|
|
|
119
119
|
| `capture_messages` | Opt-in capture of full message content on LLM-call spans (`gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions`) as GenAI-semconv JSON. Defaults to `false` — message content routinely carries sensitive data. File attachments serialize as metadata-only stubs (media type and name, never bytes), and riffer applies no size limit of its own — cap oversized attributes with the OTEL SDK attribute length limits. |
|
|
120
120
|
| `backend` | The backend riffer routes spans through. Assign `Riffer::Tracing::Otel.build` (pass `provider:` to override the global tracer provider — e.g. an in-memory provider in tests), or any object satisfying the duck-typed contract (`in_span` / `current_context` / `with_context`) to route into a non-OTEL system (e.g. Datadog APM). Defaults to `nil` — a no-op. Raises `Riffer::ArgumentError` unless the value is `nil` or responds to `in_span`. See [Tracing → Routing to a non-OpenTelemetry backend](16_TRACING.md#routing-to-a-non-opentelemetry-backend). |
|
|
121
121
|
|
|
122
|
-
### Metrics
|
|
123
|
-
|
|
124
|
-
Metrics-related global configuration lives under `config.metrics`, **independent** of `config.tracing` — each has its own kill switch and its own backend, so you can run one signal without the other. Riffer records measurements only through the backend you assign to `config.metrics.backend` — there is **no auto-detection**. OpenTelemetry is a built-in backend you opt into explicitly with `Riffer::Metrics::Otel.build`; with no backend set every measurement is a silent no-op, and riffer carries no OTEL gem dependency either way. The metrics API and SDK are separate, still-experimental (pre-1.0) gems from the traces API.
|
|
125
|
-
|
|
126
|
-
```ruby
|
|
127
|
-
Riffer.configure do |config|
|
|
128
|
-
config.metrics.enabled = ENV.fetch("RIFFER_METRICS_ENABLED", "true")
|
|
129
|
-
# Opt into OpenTelemetry (uses the global meter provider):
|
|
130
|
-
config.metrics.backend = Riffer::Metrics::Otel.build
|
|
131
|
-
end
|
|
132
|
-
```
|
|
133
|
-
|
|
134
|
-
| Option | Description |
|
|
135
|
-
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
136
|
-
| `enabled` | The kill switch, consulted on every measurement — flipping it at runtime takes effect immediately, short-circuiting to a no-op ahead of the backend. Accepts booleans or `'true'`/`'false'`/`'1'`/`'0'`. Defaults to `true`. |
|
|
137
|
-
| `backend` | The backend riffer routes measurements through. Assign `Riffer::Metrics::Otel.build` (pass `provider:` to override the global meter provider — e.g. an in-memory provider in tests), or any object responding to `record_histogram` to route into a non-OTEL system (e.g. DogStatsD). Defaults to `nil` — a no-op. Raises `Riffer::ArgumentError` unless the value is `nil` or responds to `record_histogram`. See [Metrics → Routing to a non-OpenTelemetry backend](17_METRICS.md#routing-to-a-non-opentelemetry-backend). |
|
|
138
|
-
|
|
139
122
|
### Pricing
|
|
140
123
|
|
|
141
124
|
Configure per-model token prices and riffer computes the cost of each LLM call onto its [`TokenUsage`](08_MESSAGES.md#token-usage-semantics). Riffer ships **no** price table — so an unconfigured model simply carries no cost (`token_usage.cost` is `nil`).
|
data/lib/riffer/agent/run.rb
CHANGED
|
@@ -37,33 +37,22 @@ module Riffer::Agent::Run
|
|
|
37
37
|
|
|
38
38
|
# Both +generate+ and +stream+ funnel here, so this is the single place raw
|
|
39
39
|
# +tags+ are normalized. The clean <tt>String => String</tt> map is then
|
|
40
|
-
# threaded to every span
|
|
41
|
-
#
|
|
40
|
+
# threaded to every span builder in the run as +riffer.tag.*+ and to each
|
|
41
|
+
# provider call (via +merged_model_options+) for native request-metadata
|
|
42
42
|
# mapping.
|
|
43
43
|
#--
|
|
44
44
|
#: (Riffer::Agent, ?tags: Hash[(String | Symbol), untyped]?, ?stream_yielder: Enumerator::Yielder?) -> Riffer::Agent::Response
|
|
45
45
|
def run_loop(agent, tags: {}, stream_yielder: nil)
|
|
46
46
|
tags = normalize_tags(tags)
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
response = execute_run(agent, stream_yielder, tags)
|
|
52
|
-
record_run_outcome(span, response)
|
|
53
|
-
response
|
|
54
|
-
rescue => error
|
|
55
|
-
# The backend records the exception and error status on the re-raise;
|
|
56
|
-
# error.type is the one semconv attribute it doesn't set.
|
|
57
|
-
span.set_attribute("error.type", error.class.name)
|
|
58
|
-
raise
|
|
59
|
-
end
|
|
47
|
+
Riffer::Tracing.in_span("invoke_agent #{agent.class.identifier}", attributes: run_span_attributes(agent, tags), kind: :internal) do |span|
|
|
48
|
+
response = execute_run(agent, stream_yielder, tags)
|
|
49
|
+
record_run_outcome(span, response)
|
|
50
|
+
response
|
|
60
51
|
rescue => error
|
|
61
|
-
# The
|
|
62
|
-
#
|
|
63
|
-
|
|
52
|
+
# The backend records the exception and error status on the re-raise;
|
|
53
|
+
# error.type is the one semconv attribute it doesn't set.
|
|
54
|
+
span.set_attribute("error.type", error.class.name)
|
|
64
55
|
raise
|
|
65
|
-
ensure
|
|
66
|
-
Riffer::Metrics::Instruments::OPERATION_DURATION.record(Riffer::Metrics.monotonic_now - start, attributes: run_metric_attributes(agent, error_type, tags))
|
|
67
56
|
end
|
|
68
57
|
end
|
|
69
58
|
|
|
@@ -295,8 +284,8 @@ module Riffer::Agent::Run
|
|
|
295
284
|
|
|
296
285
|
# +tags+ rides in the options hash as a curated key the providers extract for
|
|
297
286
|
# native request-metadata mapping (alongside +:structured_output+); it never
|
|
298
|
-
# reaches an SDK call verbatim. Span
|
|
299
|
-
#
|
|
287
|
+
# reaches an SDK call verbatim. Span tagging is threaded separately to each
|
|
288
|
+
# builder.
|
|
300
289
|
#--
|
|
301
290
|
#: (Riffer::Agent, ?Hash[String, String]) -> Hash[Symbol, untyped]
|
|
302
291
|
def merged_model_options(agent, tags = {})
|
|
@@ -352,19 +341,6 @@ module Riffer::Agent::Run
|
|
|
352
341
|
}.merge(tag_attributes(tags))
|
|
353
342
|
end
|
|
354
343
|
|
|
355
|
-
#--
|
|
356
|
-
#: (Riffer::Agent, String?, ?Hash[String, String]) -> Hash[String, untyped]
|
|
357
|
-
def run_metric_attributes(agent, error_type, tags = {})
|
|
358
|
-
attributes = {
|
|
359
|
-
"gen_ai.operation.name" => "invoke_agent",
|
|
360
|
-
"gen_ai.provider.name" => agent.provider.class.semconv_provider_name,
|
|
361
|
-
"gen_ai.request.model" => agent.model_name,
|
|
362
|
-
"gen_ai.agent.name" => agent.class.identifier
|
|
363
|
-
} #: Hash[String, untyped]
|
|
364
|
-
attributes["error.type"] = error_type if error_type
|
|
365
|
-
attributes.merge(tag_attributes(tags))
|
|
366
|
-
end
|
|
367
|
-
|
|
368
344
|
#--
|
|
369
345
|
#: (Hash[(String | Symbol), untyped]?) -> Hash[String, String]
|
|
370
346
|
def normalize_tags(tags)
|
data/lib/riffer/agent.rb
CHANGED
|
@@ -290,7 +290,7 @@ class Riffer::Agent
|
|
|
290
290
|
#
|
|
291
291
|
# +tags:+ is an optional flat hash of attribution tags applied to this single
|
|
292
292
|
# call: they propagate to the provider's native request-metadata field, and
|
|
293
|
-
# are stamped as +riffer.tag.*+ on every span
|
|
293
|
+
# are stamped as +riffer.tag.*+ on every span the call emits. See
|
|
294
294
|
# +docs/03_AGENTS.md+ for the per-provider mapping. The reserved key
|
|
295
295
|
# +user_id+ also maps to the provider's native user identifier where one
|
|
296
296
|
# exists.
|
data/lib/riffer/config.rb
CHANGED
|
@@ -106,47 +106,6 @@ class Riffer::Config
|
|
|
106
106
|
end
|
|
107
107
|
end
|
|
108
108
|
|
|
109
|
-
# Metrics-related global configuration, independent of +config.tracing+ so a
|
|
110
|
-
# host can run one signal without the other.
|
|
111
|
-
class Metrics
|
|
112
|
-
# Whether riffer records OTEL metric instruments; defaults to +true+, a
|
|
113
|
-
# no-op until a host wires an OTEL metrics SDK.
|
|
114
|
-
attr_reader :enabled #: bool
|
|
115
|
-
|
|
116
|
-
# The backend riffer routes measurements through; defaults to +nil+, a no-op.
|
|
117
|
-
# Riffer auto-detects no backend; assigning one is opt-in.
|
|
118
|
-
attr_reader :backend #: untyped
|
|
119
|
-
|
|
120
|
-
#--
|
|
121
|
-
#: () -> void
|
|
122
|
-
def initialize
|
|
123
|
-
@enabled = true
|
|
124
|
-
@backend = nil
|
|
125
|
-
end
|
|
126
|
-
|
|
127
|
-
# Sets the enabled flag, coercing boolean-ish values so an env-var
|
|
128
|
-
# +"false"+ (truthy in Ruby) doesn't silently keep metrics on. Raises
|
|
129
|
-
# Riffer::ArgumentError on an unrecognized value.
|
|
130
|
-
#--
|
|
131
|
-
#: (untyped) -> void
|
|
132
|
-
def enabled=(value)
|
|
133
|
-
@enabled = Riffer::Helpers::Boolean.coerce(value, attribute: "enabled")
|
|
134
|
-
end
|
|
135
|
-
|
|
136
|
-
# Sets the metrics backend riffer routes measurements through. Raises
|
|
137
|
-
# Riffer::ArgumentError unless the value is +nil+ or responds to
|
|
138
|
-
# +record_histogram+.
|
|
139
|
-
#--
|
|
140
|
-
#: (untyped) -> void
|
|
141
|
-
def backend=(value)
|
|
142
|
-
unless value.nil? || value.respond_to?(:record_histogram)
|
|
143
|
-
raise Riffer::ArgumentError, "metrics backend must respond to #record_histogram"
|
|
144
|
-
end
|
|
145
|
-
@backend = value
|
|
146
|
-
Riffer::Metrics.reset!
|
|
147
|
-
end
|
|
148
|
-
end
|
|
149
|
-
|
|
150
109
|
# Consumer-configured token pricing, keyed by +provider/model+ id. Riffer
|
|
151
110
|
# ships no price table, so an unconfigured model carries no cost.
|
|
152
111
|
class Pricing
|
|
@@ -306,9 +265,6 @@ class Riffer::Config
|
|
|
306
265
|
# Tracing-related global configuration.
|
|
307
266
|
attr_reader :tracing #: Riffer::Config::Tracing
|
|
308
267
|
|
|
309
|
-
# Metrics-related global configuration.
|
|
310
|
-
attr_reader :metrics #: Riffer::Config::Metrics
|
|
311
|
-
|
|
312
268
|
# Consumer-configured per-model token pricing.
|
|
313
269
|
attr_reader :pricing #: Riffer::Config::Pricing
|
|
314
270
|
|
|
@@ -357,7 +313,6 @@ class Riffer::Config
|
|
|
357
313
|
@tool_runtime = Riffer::Tools::Runtime::Inline.new
|
|
358
314
|
@skills = Skills.new
|
|
359
315
|
@tracing = Tracing.new
|
|
360
|
-
@metrics = Metrics.new
|
|
361
316
|
@pricing = Pricing.new
|
|
362
317
|
@message_id_strategy = :none
|
|
363
318
|
@experimental_history_healing = false
|
|
@@ -85,32 +85,15 @@ class Riffer::Guardrails::Runner
|
|
|
85
85
|
#--
|
|
86
86
|
#: (Riffer::Guardrail, untyped, messages: Array[Riffer::Messages::Base]?) -> Riffer::Guardrails::Result
|
|
87
87
|
def execute_guardrail(guardrail, data, messages:)
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
result
|
|
93
|
-
rescue => error
|
|
94
|
-
# The backend records the exception and error status on the re-raise;
|
|
95
|
-
# error.type is the one semconv attribute it doesn't set.
|
|
96
|
-
span.set_attribute("error.type", error.class.name)
|
|
97
|
-
raise
|
|
98
|
-
end
|
|
99
|
-
end
|
|
100
|
-
end
|
|
101
|
-
|
|
102
|
-
#--
|
|
103
|
-
#: (Riffer::Guardrail) { () -> Riffer::Guardrails::Result } -> Riffer::Guardrails::Result
|
|
104
|
-
def instrument_guardrail(guardrail)
|
|
105
|
-
start = Riffer::Metrics.monotonic_now
|
|
106
|
-
error_type = nil #: String?
|
|
107
|
-
begin
|
|
108
|
-
yield
|
|
88
|
+
Riffer::Tracing.in_span("execute_guardrail #{guardrail.name}", attributes: guardrail_span_attributes(guardrail), kind: :internal) do |span|
|
|
89
|
+
result = run_guardrail_phase(guardrail, data, messages: messages)
|
|
90
|
+
record_guardrail_outcome(span, result)
|
|
91
|
+
result
|
|
109
92
|
rescue => error
|
|
110
|
-
|
|
93
|
+
# The backend records the exception and error status on the re-raise;
|
|
94
|
+
# error.type is the one semconv attribute it doesn't set.
|
|
95
|
+
span.set_attribute("error.type", error.class.name)
|
|
111
96
|
raise
|
|
112
|
-
ensure
|
|
113
|
-
Riffer::Metrics::Instruments::GUARDRAIL_DURATION.record(Riffer::Metrics.monotonic_now - start, attributes: guardrail_metric_attributes(guardrail, error_type))
|
|
114
97
|
end
|
|
115
98
|
end
|
|
116
99
|
|
|
@@ -136,14 +119,6 @@ class Riffer::Guardrails::Runner
|
|
|
136
119
|
}.merge(tags.transform_keys { |key| "riffer.tag.#{key}" })
|
|
137
120
|
end
|
|
138
121
|
|
|
139
|
-
#--
|
|
140
|
-
#: (Riffer::Guardrail, String?) -> Hash[String, untyped]
|
|
141
|
-
def guardrail_metric_attributes(guardrail, error_type)
|
|
142
|
-
attributes = guardrail_span_attributes(guardrail)
|
|
143
|
-
attributes["error.type"] = error_type if error_type
|
|
144
|
-
attributes
|
|
145
|
-
end
|
|
146
|
-
|
|
147
122
|
# A block is a handled outcome, so its span status stays unset — an error
|
|
148
123
|
# span status is reserved for a raised exception.
|
|
149
124
|
#--
|
|
@@ -63,7 +63,7 @@ class Riffer::Providers::Anthropic < Riffer::Providers::Base
|
|
|
63
63
|
|
|
64
64
|
# Anthropic's only request-metadata field is metadata.user_id (opaque, no
|
|
65
65
|
# PII). It carries the reserved user_id tag; all other tags are dropped
|
|
66
|
-
# here and survive only on spans
|
|
66
|
+
# here and survive only on spans.
|
|
67
67
|
user_id = tags["user_id"]
|
|
68
68
|
params[:metadata] = {user_id: user_id} if user_id
|
|
69
69
|
|
|
@@ -46,7 +46,6 @@ class Riffer::Providers::Base
|
|
|
46
46
|
validate_normalized_messages!(messages)
|
|
47
47
|
messages = merge_consecutive_messages(messages)
|
|
48
48
|
params = build_request_params(messages, model, options)
|
|
49
|
-
tags = options[:tags] || {}
|
|
50
49
|
|
|
51
50
|
in_chat_span(model, messages, options) do |span|
|
|
52
51
|
response = execute_generate(params)
|
|
@@ -58,8 +57,6 @@ class Riffer::Providers::Base
|
|
|
58
57
|
structured_output = parse_structured_output(content) if options[:structured_output] && tool_calls.empty?
|
|
59
58
|
|
|
60
59
|
Riffer::Tracing.record_usage(span, token_usage)
|
|
61
|
-
record_token_usage_metric(model, token_usage, tags)
|
|
62
|
-
record_cost_metric(model, token_usage, tags)
|
|
63
60
|
record_finish_reason(span, finish_reason&.reason, finish_reason&.raw)
|
|
64
61
|
capture_output(span, content: content, tool_calls: tool_calls, finish_reason: finish_reason&.reason)
|
|
65
62
|
|
|
@@ -85,26 +82,17 @@ class Riffer::Providers::Base
|
|
|
85
82
|
validate_normalized_messages!(messages)
|
|
86
83
|
messages = merge_consecutive_messages(messages)
|
|
87
84
|
params = build_request_params(messages, model, options)
|
|
88
|
-
tags = options[:tags] || {}
|
|
89
85
|
|
|
90
86
|
# The enumerator body runs in its own fiber, where the fiber-local OTEL
|
|
91
87
|
# context is empty — capture here so the chat span parents to the caller's
|
|
92
|
-
# trace.
|
|
93
|
-
# the span/metric builders without re-propagation.
|
|
88
|
+
# trace.
|
|
94
89
|
trace_context = Riffer::Tracing.current_context
|
|
95
90
|
Enumerator.new do |yielder|
|
|
96
91
|
Riffer::Tracing.with_context(trace_context) do
|
|
97
92
|
in_chat_span(model, messages, options) do |span|
|
|
98
|
-
|
|
99
|
-
# it whenever either is live — metrics fire even with tracing off.
|
|
100
|
-
observe = span.recording? || Riffer::Metrics.recording?
|
|
101
|
-
sink = observe ? Riffer::Tracing::StreamRecorder.new(yielder) : yielder
|
|
93
|
+
sink = span.recording? ? Riffer::Tracing::StreamRecorder.new(yielder) : yielder
|
|
102
94
|
execute_stream(params, sink)
|
|
103
|
-
if sink.is_a?(Riffer::Tracing::StreamRecorder)
|
|
104
|
-
record_stream_outcome(span, sink)
|
|
105
|
-
record_token_usage_metric(model, sink.token_usage, tags)
|
|
106
|
-
record_cost_metric(model, sink.token_usage, tags)
|
|
107
|
-
end
|
|
95
|
+
record_stream_outcome(span, sink) if sink.is_a?(Riffer::Tracing::StreamRecorder)
|
|
108
96
|
end
|
|
109
97
|
end
|
|
110
98
|
end
|
|
@@ -225,26 +213,14 @@ class Riffer::Providers::Base
|
|
|
225
213
|
#--
|
|
226
214
|
#: [R] (String?, Array[Riffer::Messages::Base], Hash[Symbol, untyped]) { (Riffer::Tracing::Otel::Span | Riffer::Tracing::NoOp::Span) -> R } -> R
|
|
227
215
|
def in_chat_span(model, messages, options)
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
begin
|
|
232
|
-
Riffer::Tracing.in_span(model ? "chat #{model}" : "chat", attributes: chat_span_attributes(model, options), kind: :client) do |span|
|
|
233
|
-
capture_input(span, messages)
|
|
234
|
-
yield span
|
|
235
|
-
rescue => error
|
|
236
|
-
# The backend records the exception and error status on the re-raise;
|
|
237
|
-
# error.type is the one semconv attribute it doesn't set.
|
|
238
|
-
span.set_attribute("error.type", error.class.name)
|
|
239
|
-
raise
|
|
240
|
-
end
|
|
216
|
+
Riffer::Tracing.in_span(model ? "chat #{model}" : "chat", attributes: chat_span_attributes(model, options), kind: :client) do |span|
|
|
217
|
+
capture_input(span, messages)
|
|
218
|
+
yield span
|
|
241
219
|
rescue => error
|
|
242
|
-
# The
|
|
243
|
-
#
|
|
244
|
-
|
|
220
|
+
# The backend records the exception and error status on the re-raise;
|
|
221
|
+
# error.type is the one semconv attribute it doesn't set.
|
|
222
|
+
span.set_attribute("error.type", error.class.name)
|
|
245
223
|
raise
|
|
246
|
-
ensure
|
|
247
|
-
Riffer::Metrics::Instruments::OPERATION_DURATION.record(Riffer::Metrics.monotonic_now - start, attributes: chat_metric_attributes(model, error_type, tags))
|
|
248
224
|
end
|
|
249
225
|
end
|
|
250
226
|
|
|
@@ -265,54 +241,14 @@ class Riffer::Providers::Base
|
|
|
265
241
|
attributes.merge(tag_attributes(options[:tags] || {}))
|
|
266
242
|
end
|
|
267
243
|
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
def chat_metric_base_attributes(model, tags = {})
|
|
271
|
-
attributes = {
|
|
272
|
-
"gen_ai.operation.name" => "chat",
|
|
273
|
-
"gen_ai.provider.name" => self.class.semconv_provider_name
|
|
274
|
-
} #: Hash[String, untyped]
|
|
275
|
-
attributes["gen_ai.request.model"] = model if model
|
|
276
|
-
attributes.merge(tag_attributes(tags))
|
|
277
|
-
end
|
|
278
|
-
|
|
279
|
-
#--
|
|
280
|
-
#: (String?, String?, ?Hash[String, String]) -> Hash[String, untyped]
|
|
281
|
-
def chat_metric_attributes(model, error_type, tags = {})
|
|
282
|
-
attributes = chat_metric_base_attributes(model, tags)
|
|
283
|
-
attributes["error.type"] = error_type if error_type
|
|
284
|
-
attributes
|
|
285
|
-
end
|
|
286
|
-
|
|
287
|
-
# Maps normalized tags to their namespaced span/metric attribute form. An
|
|
288
|
-
# empty map yields an empty hash, so merging it is a no-op.
|
|
244
|
+
# Maps normalized tags to their namespaced span attribute form. An empty map
|
|
245
|
+
# yields an empty hash, so merging it is a no-op.
|
|
289
246
|
#--
|
|
290
247
|
#: (Hash[String, String]) -> Hash[String, String]
|
|
291
248
|
def tag_attributes(tags)
|
|
292
249
|
tags.transform_keys { |key| "riffer.tag.#{key}" }
|
|
293
250
|
end
|
|
294
251
|
|
|
295
|
-
# Per-call only — the run level would double-count an aggregate.
|
|
296
|
-
#--
|
|
297
|
-
#: (String?, Riffer::Providers::TokenUsage?, ?Hash[String, String]) -> void
|
|
298
|
-
def record_token_usage_metric(model, usage, tags = {})
|
|
299
|
-
return unless usage
|
|
300
|
-
|
|
301
|
-
base = chat_metric_base_attributes(model, tags)
|
|
302
|
-
Riffer::Metrics::Instruments::TOKEN_USAGE.record(usage.input_tokens, attributes: base.merge("gen_ai.token.type" => "input"))
|
|
303
|
-
Riffer::Metrics::Instruments::TOKEN_USAGE.record(usage.output_tokens, attributes: base.merge("gen_ai.token.type" => "output"))
|
|
304
|
-
end
|
|
305
|
-
|
|
306
|
-
# Per-call only — the run level would double-count an aggregate.
|
|
307
|
-
#--
|
|
308
|
-
#: (String?, Riffer::Providers::TokenUsage?, ?Hash[String, String]) -> void
|
|
309
|
-
def record_cost_metric(model, usage, tags = {})
|
|
310
|
-
cost = usage&.cost
|
|
311
|
-
return unless cost
|
|
312
|
-
|
|
313
|
-
Riffer::Metrics::Instruments::COST.record(cost, attributes: chat_metric_base_attributes(model, tags))
|
|
314
|
-
end
|
|
315
|
-
|
|
316
252
|
#--
|
|
317
253
|
#: ((Riffer::Tracing::Otel::Span | Riffer::Tracing::NoOp::Span), Symbol?, String?) -> void
|
|
318
254
|
def record_finish_reason(span, reason, raw)
|
data/lib/riffer/tools/runtime.rb
CHANGED
|
@@ -23,7 +23,7 @@ class Riffer::Tools::Runtime
|
|
|
23
23
|
# Each Runner worker runs in its own thread/fiber, where the OTEL context
|
|
24
24
|
# starts empty — capture here so the execute_tool span parents correctly.
|
|
25
25
|
# tags are an ordinary local captured in the block, so they reach the
|
|
26
|
-
# worker's span
|
|
26
|
+
# worker's span without re-propagation.
|
|
27
27
|
trace_context = Riffer::Tracing.current_context
|
|
28
28
|
@runner.map(tool_calls, context: context) do |tool_call|
|
|
29
29
|
Riffer::Tracing.with_context(trace_context) do
|
|
@@ -61,22 +61,12 @@ class Riffer::Tools::Runtime
|
|
|
61
61
|
#--
|
|
62
62
|
#: (Riffer::Messages::Assistant::ToolCall, ?Hash[String, String]) { () -> Riffer::Tools::Response } -> [Riffer::Messages::Assistant::ToolCall, Riffer::Tools::Response]
|
|
63
63
|
def instrument_tool_call(tool_call, tags = {})
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
response = yield
|
|
69
|
-
record_tool_outcome(span, response)
|
|
70
|
-
response
|
|
71
|
-
end
|
|
72
|
-
error_type = result.error_type&.to_s
|
|
73
|
-
[tool_call, result] #: [Riffer::Messages::Assistant::ToolCall, Riffer::Tools::Response]
|
|
74
|
-
rescue => error
|
|
75
|
-
error_type = error.class.name #: String?
|
|
76
|
-
raise
|
|
77
|
-
ensure
|
|
78
|
-
Riffer::Metrics::Instruments::OPERATION_DURATION.record(Riffer::Metrics.monotonic_now - start, attributes: tool_metric_attributes(tool_call, error_type, tags))
|
|
64
|
+
result = in_tool_span(tool_call, tags) do |span|
|
|
65
|
+
response = yield
|
|
66
|
+
record_tool_outcome(span, response)
|
|
67
|
+
response
|
|
79
68
|
end
|
|
69
|
+
[tool_call, result] #: [Riffer::Messages::Assistant::ToolCall, Riffer::Tools::Response]
|
|
80
70
|
end
|
|
81
71
|
|
|
82
72
|
#--
|
|
@@ -138,19 +128,8 @@ class Riffer::Tools::Runtime
|
|
|
138
128
|
}.merge(tag_attributes(tags))
|
|
139
129
|
end
|
|
140
130
|
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
def tool_metric_attributes(tool_call, error_type, tags = {})
|
|
144
|
-
attributes = {
|
|
145
|
-
"gen_ai.operation.name" => "execute_tool",
|
|
146
|
-
"gen_ai.tool.name" => tool_call.name
|
|
147
|
-
} #: Hash[String, untyped]
|
|
148
|
-
attributes["error.type"] = error_type if error_type
|
|
149
|
-
attributes.merge(tag_attributes(tags))
|
|
150
|
-
end
|
|
151
|
-
|
|
152
|
-
# Maps normalized tags to their namespaced span/metric attribute form. An
|
|
153
|
-
# empty map yields an empty hash, so merging it is a no-op.
|
|
131
|
+
# Maps normalized tags to their namespaced span attribute form. An empty map
|
|
132
|
+
# yields an empty hash, so merging it is a no-op.
|
|
154
133
|
#--
|
|
155
134
|
#: (Hash[String, String]) -> Hash[String, String]
|
|
156
135
|
def tag_attributes(tags)
|
data/lib/riffer/version.rb
CHANGED
|
@@ -1,13 +1,10 @@
|
|
|
1
|
-
# Minimal signatures for the `opentelemetry-api`
|
|
2
|
-
#
|
|
1
|
+
# Minimal signatures for the `opentelemetry-api` gem (undeclared, runtime
|
|
2
|
+
# detected), which ships no RBS.
|
|
3
3
|
#
|
|
4
|
-
# Only the surface used by Riffer::Tracing::Otel
|
|
5
|
-
# declared.
|
|
4
|
+
# Only the surface used by Riffer::Tracing::Otel is declared.
|
|
6
5
|
module OpenTelemetry
|
|
7
6
|
def self.tracer_provider: () -> untyped
|
|
8
7
|
|
|
9
|
-
def self.meter_provider: () -> untyped
|
|
10
|
-
|
|
11
8
|
class Context
|
|
12
9
|
def self.current: () -> OpenTelemetry::Context
|
|
13
10
|
|
|
@@ -21,8 +21,8 @@ module Riffer::Agent::Run
|
|
|
21
21
|
|
|
22
22
|
# Both +generate+ and +stream+ funnel here, so this is the single place raw
|
|
23
23
|
# +tags+ are normalized. The clean <tt>String => String</tt> map is then
|
|
24
|
-
# threaded to every span
|
|
25
|
-
#
|
|
24
|
+
# threaded to every span builder in the run as +riffer.tag.*+ and to each
|
|
25
|
+
# provider call (via +merged_model_options+) for native request-metadata
|
|
26
26
|
# mapping.
|
|
27
27
|
# --
|
|
28
28
|
# : (Riffer::Agent, ?tags: Hash[(String | Symbol), untyped]?, ?stream_yielder: Enumerator::Yielder?) -> Riffer::Agent::Response
|
|
@@ -86,8 +86,8 @@ module Riffer::Agent::Run
|
|
|
86
86
|
|
|
87
87
|
# +tags+ rides in the options hash as a curated key the providers extract for
|
|
88
88
|
# native request-metadata mapping (alongside +:structured_output+); it never
|
|
89
|
-
# reaches an SDK call verbatim. Span
|
|
90
|
-
#
|
|
89
|
+
# reaches an SDK call verbatim. Span tagging is threaded separately to each
|
|
90
|
+
# builder.
|
|
91
91
|
# --
|
|
92
92
|
# : (Riffer::Agent, ?Hash[String, String]) -> Hash[Symbol, untyped]
|
|
93
93
|
def merged_model_options: (Riffer::Agent, ?Hash[String, String]) -> Hash[Symbol, untyped]
|
|
@@ -114,10 +114,6 @@ module Riffer::Agent::Run
|
|
|
114
114
|
# : (Riffer::Agent, ?Hash[String, String]) -> Hash[String, untyped]
|
|
115
115
|
def run_span_attributes: (Riffer::Agent, ?Hash[String, String]) -> Hash[String, untyped]
|
|
116
116
|
|
|
117
|
-
# --
|
|
118
|
-
# : (Riffer::Agent, String?, ?Hash[String, String]) -> Hash[String, untyped]
|
|
119
|
-
def run_metric_attributes: (Riffer::Agent, String?, ?Hash[String, String]) -> Hash[String, untyped]
|
|
120
|
-
|
|
121
117
|
# --
|
|
122
118
|
# : (Hash[(String | Symbol), untyped]?) -> Hash[String, String]
|
|
123
119
|
def normalize_tags: (Hash[String | Symbol, untyped]?) -> Hash[String, String]
|
|
@@ -215,7 +215,7 @@ class Riffer::Agent
|
|
|
215
215
|
#
|
|
216
216
|
# +tags:+ is an optional flat hash of attribution tags applied to this single
|
|
217
217
|
# call: they propagate to the provider's native request-metadata field, and
|
|
218
|
-
# are stamped as +riffer.tag.*+ on every span
|
|
218
|
+
# are stamped as +riffer.tag.*+ on every span the call emits. See
|
|
219
219
|
# +docs/03_AGENTS.md+ for the per-provider mapping. The reserved key
|
|
220
220
|
# +user_id+ also maps to the provider's native user identifier where one
|
|
221
221
|
# exists.
|
|
@@ -137,36 +137,6 @@ class Riffer::Config
|
|
|
137
137
|
def backend=: (untyped) -> void
|
|
138
138
|
end
|
|
139
139
|
|
|
140
|
-
# Metrics-related global configuration, independent of +config.tracing+ so a
|
|
141
|
-
# host can run one signal without the other.
|
|
142
|
-
class Metrics
|
|
143
|
-
# Whether riffer records OTEL metric instruments; defaults to +true+, a
|
|
144
|
-
# no-op until a host wires an OTEL metrics SDK.
|
|
145
|
-
attr_reader enabled: bool
|
|
146
|
-
|
|
147
|
-
# The backend riffer routes measurements through; defaults to +nil+, a no-op.
|
|
148
|
-
# Riffer auto-detects no backend; assigning one is opt-in.
|
|
149
|
-
attr_reader backend: untyped
|
|
150
|
-
|
|
151
|
-
# --
|
|
152
|
-
# : () -> void
|
|
153
|
-
def initialize: () -> void
|
|
154
|
-
|
|
155
|
-
# Sets the enabled flag, coercing boolean-ish values so an env-var
|
|
156
|
-
# +"false"+ (truthy in Ruby) doesn't silently keep metrics on. Raises
|
|
157
|
-
# Riffer::ArgumentError on an unrecognized value.
|
|
158
|
-
# --
|
|
159
|
-
# : (untyped) -> void
|
|
160
|
-
def enabled=: (untyped) -> void
|
|
161
|
-
|
|
162
|
-
# Sets the metrics backend riffer routes measurements through. Raises
|
|
163
|
-
# Riffer::ArgumentError unless the value is +nil+ or responds to
|
|
164
|
-
# +record_histogram+.
|
|
165
|
-
# --
|
|
166
|
-
# : (untyped) -> void
|
|
167
|
-
def backend=: (untyped) -> void
|
|
168
|
-
end
|
|
169
|
-
|
|
170
140
|
# Consumer-configured token pricing, keyed by +provider/model+ id. Riffer
|
|
171
141
|
# ships no price table, so an unconfigured model carries no cost.
|
|
172
142
|
class Pricing
|
|
@@ -276,9 +246,6 @@ class Riffer::Config
|
|
|
276
246
|
# Tracing-related global configuration.
|
|
277
247
|
attr_reader tracing: Riffer::Config::Tracing
|
|
278
248
|
|
|
279
|
-
# Metrics-related global configuration.
|
|
280
|
-
attr_reader metrics: Riffer::Config::Metrics
|
|
281
|
-
|
|
282
249
|
# Consumer-configured per-model token pricing.
|
|
283
250
|
attr_reader pricing: Riffer::Config::Pricing
|
|
284
251
|
|