nitro_intelligence 4.1.1 → 4.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/docs/CHANGELOG.md +15 -0
- data/docs/README.md +69 -1
- data/lib/nitro_intelligence/client/observers/langfuse_observer.rb +30 -26
- data/lib/nitro_intelligence/langfuse_extension.rb +52 -0
- data/lib/nitro_intelligence/langfuse_tracer_provider.rb +11 -2
- data/lib/nitro_intelligence/media/image_generation.rb +6 -1
- data/lib/nitro_intelligence/observability/batch_span_processor.rb +200 -0
- data/lib/nitro_intelligence/observability/project_client.rb +23 -0
- data/lib/nitro_intelligence/observability/project_client_registry.rb +6 -2
- data/lib/nitro_intelligence/observation.rb +132 -0
- data/lib/nitro_intelligence/reporter.rb +16 -4
- data/lib/nitro_intelligence/tracer.rb +87 -0
- data/lib/nitro_intelligence/version.rb +1 -1
- data/lib/nitro_intelligence.rb +9 -4
- metadata +24 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4f03c055275bc0a35eda96d094085e42cefcedf07a9b02e621ecf1ba88bb32f7
|
|
4
|
+
data.tar.gz: f11cf2e9e924eff3fd859f9d43a99eaf9993aa11f83308df4354ce717db32627
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: bbd1c842b1ff8fadc8d86d2090cd8945ebf71cb7354bb815866d0082a383bb4c07113fd3323db420f28db9f7761b2e588cac2c7dd9fd4f294357c86bcc40b53e
|
|
7
|
+
data.tar.gz: 8f0a724e2c3d44c0370e333e9dd65842dd4a318c47aa5039feb7c8070b062410ec91337fb305b7fb5133afbafac510f4e6293c59e7c2b47dc9a305357eeacd32
|
data/docs/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,21 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [4.2.0](https://github.com/powerhome/nitro-intelligence.rb/compare/v4.1.1...v4.2.0) (2026-11-10)
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
* let applications trace their own steps alongside the gem's requests ([#198](https://github.com/powerhome/nitro-intelligence.rb/issues/198)) ([dd45726](https://github.com/powerhome/nitro-intelligence.rb/commit/dd457262545752064030755591cd40b265b1cae3))
|
|
14
|
+
* let Reporter#score take observation_id, comment and metadata to ([dd45726](https://github.com/powerhome/nitro-intelligence.rb/commit/dd457262545752064030755591cd40b265b1cae3))
|
|
15
|
+
* **observability:** add project client flush ([#206](https://github.com/powerhome/nitro-intelligence.rb/issues/206)) ([0303034](https://github.com/powerhome/nitro-intelligence.rb/commit/0303034e5586e3fdfa0704be1899b93cbf8b90ce))
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
### Fixed
|
|
19
|
+
|
|
20
|
+
* **observability:** share one project client registry and one client per ([0303034](https://github.com/powerhome/nitro-intelligence.rb/commit/0303034e5586e3fdfa0704be1899b93cbf8b90ce))
|
|
21
|
+
* use the final image when Gemini also returns thought images ([#205](https://github.com/powerhome/nitro-intelligence.rb/issues/205)) ([1ce6b50](https://github.com/powerhome/nitro-intelligence.rb/commit/1ce6b501e7d374de6b39a1d7cabe3314de69c01c))
|
|
22
|
+
|
|
8
23
|
## [4.1.1](https://github.com/powerhome/nitro-intelligence.rb/compare/v4.1.0...v4.1.1) (2026-11-05)
|
|
9
24
|
|
|
10
25
|
|
data/docs/README.md
CHANGED
|
@@ -275,7 +275,7 @@ client = NitroIntelligence::Client.new
|
|
|
275
275
|
result = client.generate_image(message: "Create an image of a bear installing a window.")
|
|
276
276
|
```
|
|
277
277
|
|
|
278
|
-
`result` is a `NitroIntelligence::ImageGeneration` object, and the generated image can be accessed via `result.generated_image` which returns a `NitroIntelligence::Image`.
|
|
278
|
+
`result` is a `NitroIntelligence::ImageGeneration` object, and the generated image can be accessed via `result.generated_image` which returns a `NitroIntelligence::Image`. Gemini image models that think can return their interim drafts alongside the final render; `generated_image` is always the final one.
|
|
279
279
|
|
|
280
280
|
With observability enabled, `result.trace_id` and `result.observation_id` identify the trace and generation observation. Both are `nil` without observability.
|
|
281
281
|
|
|
@@ -343,6 +343,25 @@ client.chat(
|
|
|
343
343
|
|
|
344
344
|
If no `prompt_label` or `prompt_version` is provided, the 'production' prompt is used by default.
|
|
345
345
|
|
|
346
|
+
### Exporting Completed Observations
|
|
347
|
+
|
|
348
|
+
Observations are exported in the background. Use the same shared project client for tracing and flushing when a job needs to export completed observations before submitting their IDs for human review:
|
|
349
|
+
|
|
350
|
+
```ruby
|
|
351
|
+
project_client = NitroIntelligence.project_client_registry.fetch("my-feature-project")
|
|
352
|
+
raise "Observation export failed or timed out" unless project_client.flush(timeout: 30)
|
|
353
|
+
|
|
354
|
+
# Submit the completed observations' IDs to the annotation queue here.
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
`flush` waits only for this project's observations queued or in flight when called. Observations completed later do not extend the wait. The client remains usable, and active observations are not ended.
|
|
358
|
+
|
|
359
|
+
It returns `true` when those exports succeed or nothing needs exporting. It returns `false` if exporting fails or cannot start, an observation being flushed is dropped, the wait times out, or shutdown prevents completion. Calls made during or after shutdown return `false`. Earlier export failures are not reported.
|
|
360
|
+
|
|
361
|
+
The timeout defaults to 30 seconds; exports continue after a caller times out. With `timeout: 0`, a running client succeeds only if nothing is queued or in flight.
|
|
362
|
+
|
|
363
|
+
It does not flush queued scores or wait for evaluators, and successful export does not guarantee immediate visibility in Cerebro or annotation queues. Reserve `NitroIntelligence.project_client_registry.shutdown_all` for process teardown. See the [ProjectClient#flush API reference](https://www.rubydoc.info/gems/nitro_intelligence/NitroIntelligence/Observability/ProjectClient#flush-instance_method) for details.
|
|
364
|
+
|
|
346
365
|
### Prompt Fallbacks
|
|
347
366
|
|
|
348
367
|
Pass `prompt_fallback_name` to name a second prompt to use when the one you asked for isn't there. The prompt in `prompt_name` is tried first; if it doesn't exist, or its lookup fails, the fallback is used instead:
|
|
@@ -521,6 +540,46 @@ Two things this does not cover:
|
|
|
521
540
|
* **The gateway's redaction is by field, and misses some.** It replaces the fields it knows about, such as `content` and `reasoning_content`, rather than the whole response, so a field a provider adds beyond those is still logged. Reasoning that a provider returns under `provider_specific_fields` is one such field today; see [powerhome/nitro-intelligence#704](https://github.com/powerhome/nitro-intelligence/issues/704).
|
|
522
541
|
* **Prompts you fetch from Cerebro are unaffected.** Masking applies to traces, not to prompt management.
|
|
523
542
|
|
|
543
|
+
### Tracing Application Steps
|
|
544
|
+
|
|
545
|
+
Each observed request is a trace of its own by default. A feature whose work is more than one model call, such as a search that embeds a question, queries an index, reranks the results and filters them by permission, can record that work as one trace with `NitroIntelligence::Tracer`: its own steps as observations, and the gem's requests nested among them.
|
|
546
|
+
|
|
547
|
+
```ruby
|
|
548
|
+
tracer = NitroIntelligence::Tracer.new(observability_project_slug: "page-finder")
|
|
549
|
+
client = NitroIntelligence::Client.new(observability_project_slug: "page-finder")
|
|
550
|
+
|
|
551
|
+
pages = tracer.trace("Page Finder search", input: { question: }, user_id: current_user.id.to_s, session_id:) do |root_observation|
|
|
552
|
+
query = client.chat(message: question) # recorded inside the trace
|
|
553
|
+
|
|
554
|
+
hits = root_observation.observe("Elasticsearch", as_type: :retriever, input: { k: 50, boost: 2 }) do |step|
|
|
555
|
+
search(query).tap { |hits| step.output = hits.map { |hit| { id: hit.page_id, score: hit.score } } }
|
|
556
|
+
end
|
|
557
|
+
|
|
558
|
+
allowed = root_observation.observe("Permission filter", input: { candidates: hits.size }) do |step|
|
|
559
|
+
permitted(hits).tap { |pages| step.output = { allowed: pages.size } }
|
|
560
|
+
end
|
|
561
|
+
|
|
562
|
+
root_observation.output = allowed.map(&:page_id)
|
|
563
|
+
@trace_id = root_observation.trace_id # to score the search when the user clicks a result
|
|
564
|
+
allowed
|
|
565
|
+
end
|
|
566
|
+
```
|
|
567
|
+
|
|
568
|
+
`Tracer#trace` records its block as a trace with the given name, user, session, tags and metadata, and the environment and release the gem's own requests record. `user_id` defaults to `observability_user_id`, as it does for a request. It yields the trace's root observation: the input and output set on it are the trace's input and output too, while metadata set on it is recorded on that observation only. `trace_seed:` gives the trace the ID `NitroIntelligence::Trace.create_id` gives for the seed; otherwise the ID is random, and `root_observation.trace_id` reads it.
|
|
569
|
+
|
|
570
|
+
`Observation#observe` records a step inside the trace, or inside another step, around a block. `as_type` is a Langfuse [observation type](https://langfuse.com/docs/observability/features/observation-types): `:span` by default, or `:retriever`, `:tool`, `:agent`, `:chain`, `:evaluator`, `:guardrail`, `:embedding` or `:generation`. `Observation#event` records a point in time, such as the results a user was shown. An exception raised inside a block is recorded on its step at level `ERROR`, and re-raised.
|
|
571
|
+
|
|
572
|
+
The gem's requests made inside the block are recorded as observations under the step that is open, not as traces of their own, when:
|
|
573
|
+
|
|
574
|
+
* the client records in the same Cerebro project as the tracer. A request recorded in another project is a trace of its own there, as it would be outside the block.
|
|
575
|
+
* the request passes no `trace_seed`. A seeded request is recorded in the seed's trace, as it is anywhere.
|
|
576
|
+
|
|
577
|
+
Such a request records its own observation only: the trace's name, input, output, user, session and tags are the ones the tracer was given, and the request's `trace_name`, `user_id`, `session_id` and `tags` are not recorded. Its `metadata` is still recorded on its observation and sent to the inference gateway.
|
|
578
|
+
|
|
579
|
+
A trace of its own started inside the block (a request to another project, a seeded request, or another `Tracer#trace`) neither takes the open trace's user, session, tags and metadata nor changes them. It takes only what the application propagated before opening the trace, such as with `Langfuse.propagate_attributes`, as it would outside the block.
|
|
580
|
+
|
|
581
|
+
A project that masks content records the input and output of every step and event as `[masked]`, and a failure's error class without its message, as it does for the gem's requests. Metadata, names, user, session and tags are not masked.
|
|
582
|
+
|
|
524
583
|
### Scoring
|
|
525
584
|
|
|
526
585
|
You can use `NitroIntelligence::Reporter` to evaluate existing traces. Calling `NitroIntelligence::Reporter#score` lets you attach metrics to a trace in the observability platform.
|
|
@@ -543,6 +602,15 @@ trace_id = NitroIntelligence::Trace.create_id(seed: document_id)
|
|
|
543
602
|
reporter.score(name: "precision", value: 0.5, trace_id:)
|
|
544
603
|
```
|
|
545
604
|
|
|
605
|
+
#### An Observation, Later
|
|
606
|
+
|
|
607
|
+
A score can be recorded from a later request than the one that made the trace, such as a click on a search result, given the trace's ID. Pass `observation_id:` to score one observation in the trace, as `Observation#id` gives it, rather than the trace as a whole. `comment:` and `metadata:` are recorded with the score.
|
|
608
|
+
|
|
609
|
+
```ruby
|
|
610
|
+
reporter = NitroIntelligence::Reporter.new(observability_project_slug: "page-finder")
|
|
611
|
+
reporter.score(trace_id: params[:trace_id], name: "clicked", value: true, data_type: :boolean, metadata: { position: 3 })
|
|
612
|
+
```
|
|
613
|
+
|
|
546
614
|
#### Using different data types for scoring
|
|
547
615
|
|
|
548
616
|
By default, scores submitted on a trace are numeric, but you can use the `data_type` keyword to change it to `:boolean` or `:categorical`. For example:
|
|
@@ -22,7 +22,7 @@ module NitroIntelligence
|
|
|
22
22
|
@project_client = project_client
|
|
23
23
|
end
|
|
24
24
|
|
|
25
|
-
def observe(operation_name, type:, parameters:, trace_name:, prompt: nil, input: nil)
|
|
25
|
+
def observe(operation_name, type:, parameters:, trace_name:, prompt: nil, input: nil, &)
|
|
26
26
|
metadata = parameters[:metadata]
|
|
27
27
|
seed = parameters[:trace_seed]
|
|
28
28
|
trace_id = NitroIntelligence::Trace.create_id(seed:) if seed.present?
|
|
@@ -34,25 +34,19 @@ module NitroIntelligence
|
|
|
34
34
|
end
|
|
35
35
|
|
|
36
36
|
metadata = metadata.transform_values(&:to_s)
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
result, trace_attributes = observe_failures(generation) { yield(generation) }
|
|
52
|
-
record_result(generation, trace_attributes)
|
|
53
|
-
|
|
54
|
-
result
|
|
55
|
-
end
|
|
37
|
+
# Inside a trace the application opened, the trace's name, input, output, user, session and
|
|
38
|
+
# tags are the application's, so the request records only its own observation.
|
|
39
|
+
nested = trace_id.nil? && @project_client.observability_client.observing?
|
|
40
|
+
request = { operation_name:, type:, trace_id:, trace_name:, prompt:, input:, metadata:, nested: }
|
|
41
|
+
request[:model] = parameters[:model]
|
|
42
|
+
|
|
43
|
+
# Propagating sets the attributes on the active span too, which inside an application's
|
|
44
|
+
# trace is the application's own observation.
|
|
45
|
+
return record_request(**request, &) if nested
|
|
46
|
+
|
|
47
|
+
propagated = propagated_attributes(parameters, metadata)
|
|
48
|
+
LangfuseExtension.in_new_trace do
|
|
49
|
+
Langfuse.propagate_attributes(**propagated) { record_request(**request, &) }
|
|
56
50
|
end
|
|
57
51
|
end
|
|
58
52
|
|
|
@@ -62,6 +56,21 @@ module NitroIntelligence
|
|
|
62
56
|
@project_client.project
|
|
63
57
|
end
|
|
64
58
|
|
|
59
|
+
def record_request(operation_name:, type:, trace_id:, trace_name:, prompt:, input:, model:, metadata:, nested:) # rubocop:disable Metrics/ParameterLists
|
|
60
|
+
@project_client.observability_client.observe(
|
|
61
|
+
operation_name, as_type: type, trace_id:, environment: NitroIntelligence.environment.to_s, model:, metadata:
|
|
62
|
+
) do |generation|
|
|
63
|
+
generation.update_trace(name: trace_name) unless nested
|
|
64
|
+
generation.update({ prompt: { name: prompt.name, version: prompt.version } }) if prompt
|
|
65
|
+
record_input(generation, input)
|
|
66
|
+
|
|
67
|
+
result, trace_attributes = observe_failures(generation) { yield(generation) }
|
|
68
|
+
record_result(generation, trace_attributes)
|
|
69
|
+
|
|
70
|
+
result
|
|
71
|
+
end
|
|
72
|
+
end
|
|
73
|
+
|
|
65
74
|
# Applied once the response is in hand, so the observation reflects what came
|
|
66
75
|
# back rather than what was asked for. Each attribute is set only when the
|
|
67
76
|
# handler supplied it: an observation that records nothing is better than one
|
|
@@ -85,8 +94,6 @@ module NitroIntelligence
|
|
|
85
94
|
end
|
|
86
95
|
|
|
87
96
|
record_status(generation, trace_attributes)
|
|
88
|
-
|
|
89
|
-
generation.update_trace(input: trace_attributes[:input], output: trace_attributes[:output])
|
|
90
97
|
end
|
|
91
98
|
|
|
92
99
|
def masked(trace_attributes)
|
|
@@ -119,10 +126,7 @@ module NitroIntelligence
|
|
|
119
126
|
def record_input(generation, input)
|
|
120
127
|
return if input.blank?
|
|
121
128
|
|
|
122
|
-
input = project.mask(input)
|
|
123
|
-
|
|
124
|
-
generation.input = input
|
|
125
|
-
generation.update_trace(input:)
|
|
129
|
+
generation.input = project.mask(input)
|
|
126
130
|
end
|
|
127
131
|
|
|
128
132
|
# `session_id` and `tags` are only forwarded when set so that callers who
|
|
@@ -13,6 +13,33 @@ require "nitro_intelligence/langfuse_tracer_provider"
|
|
|
13
13
|
module NitroIntelligence
|
|
14
14
|
# @api private
|
|
15
15
|
class LangfuseExtension
|
|
16
|
+
# Holds the client recording the {Observation} application code has open, so that observations
|
|
17
|
+
# started inside it nest under it only when they are recorded in the same project.
|
|
18
|
+
ACTIVE_OBSERVATION_KEY = OpenTelemetry::Context.create_key("nitro-intelligence-active-observation")
|
|
19
|
+
|
|
20
|
+
# Holds the context the application had before it opened the trace it has open now.
|
|
21
|
+
TRACE_BASE_KEY = OpenTelemetry::Context.create_key("nitro-intelligence-trace-base")
|
|
22
|
+
|
|
23
|
+
# What a trace the application opens adds to the context, besides its active span.
|
|
24
|
+
TRACE_KEYS = [*Langfuse::Propagation::CONTEXT_KEYS.values, ACTIVE_OBSERVATION_KEY, TRACE_BASE_KEY].freeze
|
|
25
|
+
|
|
26
|
+
# Runs the block for a trace of its own. Inside a trace the application opened, the block runs
|
|
27
|
+
# without that trace's active span and propagated attributes, as it would outside the trace:
|
|
28
|
+
# `Langfuse.propagate_attributes` writes onto the active span and merges into what the context
|
|
29
|
+
# already propagates, so a trace started inside another would otherwise change it and inherit
|
|
30
|
+
# from it.
|
|
31
|
+
def self.in_new_trace(&)
|
|
32
|
+
context = OpenTelemetry::Context.current
|
|
33
|
+
base = context.value(TRACE_BASE_KEY)
|
|
34
|
+
unless base.nil?
|
|
35
|
+
context = OpenTelemetry::Trace.context_with_span(OpenTelemetry::Trace.current_span(base),
|
|
36
|
+
parent_context: context)
|
|
37
|
+
context = TRACE_KEYS.reduce(context) { |restored, key| restored.set_value(key, base.value(key)) }
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
OpenTelemetry::Context.with_current(context.set_value(TRACE_BASE_KEY, context), &)
|
|
41
|
+
end
|
|
42
|
+
|
|
16
43
|
attr_reader :config
|
|
17
44
|
|
|
18
45
|
def initialize
|
|
@@ -24,6 +51,11 @@ module NitroIntelligence
|
|
|
24
51
|
@tracer_provider = NitroIntelligence::LangfuseTracerProvider.new(config)
|
|
25
52
|
end
|
|
26
53
|
|
|
54
|
+
# @api private
|
|
55
|
+
def flush(timeout: 30)
|
|
56
|
+
@tracer_provider.force_flush(timeout:)
|
|
57
|
+
end
|
|
58
|
+
|
|
27
59
|
def shutdown(timeout: 30)
|
|
28
60
|
@client.shutdown
|
|
29
61
|
@tracer_provider.shutdown(timeout:)
|
|
@@ -47,6 +79,20 @@ module NitroIntelligence
|
|
|
47
79
|
)
|
|
48
80
|
end
|
|
49
81
|
|
|
82
|
+
# Whether application code has an observation open in this client, under which an observation
|
|
83
|
+
# started without a trace ID or parent nests.
|
|
84
|
+
def observing?
|
|
85
|
+
active_observation_client.equal?(self)
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
# Runs the block with `observation` as the active span, so that observations this client starts
|
|
89
|
+
# inside it, the gem's own included, nest under it.
|
|
90
|
+
def within(observation, &)
|
|
91
|
+
context = OpenTelemetry::Trace.context_with_span(observation.otel_span)
|
|
92
|
+
context = context.set_value(ACTIVE_OBSERVATION_KEY, self)
|
|
93
|
+
OpenTelemetry::Context.with_current(context, &)
|
|
94
|
+
end
|
|
95
|
+
|
|
50
96
|
def start_observation(name, attrs = {}, as_type: :span, trace_id: nil, parent_span_context: nil, start_time: nil, # rubocop:disable Metrics/ParameterLists
|
|
51
97
|
skip_validation: false)
|
|
52
98
|
parent_span_context = Langfuse.send(:resolve_trace_context, trace_id, parent_span_context)
|
|
@@ -87,5 +133,11 @@ module NitroIntelligence
|
|
|
87
133
|
|
|
88
134
|
observation.send(:run_in_context, &block)
|
|
89
135
|
end
|
|
136
|
+
|
|
137
|
+
private
|
|
138
|
+
|
|
139
|
+
def active_observation_client
|
|
140
|
+
OpenTelemetry::Context.value(ACTIVE_OBSERVATION_KEY)
|
|
141
|
+
end
|
|
90
142
|
end
|
|
91
143
|
end
|
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
require "nitro_intelligence/observability/batch_span_processor"
|
|
2
|
+
|
|
1
3
|
module NitroIntelligence
|
|
2
4
|
# This is an adaptation of https://github.com/simplepractice/langfuse-rb/blob/main/lib/langfuse/otel_setup.rb
|
|
3
5
|
# that lets users create multiple Langfuse clients in the same process.
|
|
@@ -16,6 +18,13 @@ module NitroIntelligence
|
|
|
16
18
|
@tracer_provider.tracer("langfuse-rb", Langfuse::VERSION)
|
|
17
19
|
end
|
|
18
20
|
|
|
21
|
+
# @api private
|
|
22
|
+
def force_flush(timeout: 30)
|
|
23
|
+
# Langfuse's other span processor only adds attributes; its flush is a no-op.
|
|
24
|
+
# Calling our processor directly also avoids the provider's unbounded flush mutex.
|
|
25
|
+
@batch_processor.force_flush(timeout:)
|
|
26
|
+
end
|
|
27
|
+
|
|
19
28
|
def shutdown(timeout: 30)
|
|
20
29
|
@tracer_provider.shutdown(timeout:)
|
|
21
30
|
end
|
|
@@ -50,7 +59,7 @@ module NitroIntelligence
|
|
|
50
59
|
compression: "gzip"
|
|
51
60
|
)
|
|
52
61
|
|
|
53
|
-
|
|
62
|
+
@batch_processor = Observability::BatchSpanProcessor.new(
|
|
54
63
|
exporter,
|
|
55
64
|
max_queue_size: config.batch_size * 2,
|
|
56
65
|
schedule_delay: config.flush_interval * 1000,
|
|
@@ -58,7 +67,7 @@ module NitroIntelligence
|
|
|
58
67
|
)
|
|
59
68
|
|
|
60
69
|
tracer_provider = OpenTelemetry::SDK::Trace::TracerProvider.new(id_generator: self)
|
|
61
|
-
tracer_provider.add_span_processor(
|
|
70
|
+
tracer_provider.add_span_processor(@batch_processor)
|
|
62
71
|
span_processor = Langfuse::SpanProcessor.new(config:)
|
|
63
72
|
tracer_provider.add_span_processor(span_processor)
|
|
64
73
|
|
|
@@ -64,8 +64,13 @@ module NitroIntelligence
|
|
|
64
64
|
end
|
|
65
65
|
|
|
66
66
|
# @api private
|
|
67
|
+
#
|
|
68
|
+
# Takes the last entry in `message.images`. Gemini image models that think put their interim
|
|
69
|
+
# thought images ahead of the final render in the same list, and Google documents the last image
|
|
70
|
+
# as the final one.
|
|
67
71
|
def parse_file(chat_completion)
|
|
68
|
-
|
|
72
|
+
images = chat_completion.choices.first&.message.to_h[:images]
|
|
73
|
+
base64_string = Array(images).last&.dig(:image_url, :url)
|
|
69
74
|
|
|
70
75
|
return unless base64_string
|
|
71
76
|
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
module NitroIntelligence
|
|
2
|
+
module Observability
|
|
3
|
+
# Coordinates flush with the existing export worker instead of exporting a separate
|
|
4
|
+
# queue snapshot. OpenTelemetry Ruby 1.13.1's force_flush misses in-flight batches
|
|
5
|
+
# and waits for export locks without a deadline. Keep its queue, exporter and metrics;
|
|
6
|
+
# wake its worker and wait for the work captured at the call's start with a deadline.
|
|
7
|
+
# The OTLP exporter sends synchronously and its force_flush is a no-op, so finishing
|
|
8
|
+
# the worker's export also finishes that batch's delivery attempt.
|
|
9
|
+
#
|
|
10
|
+
# Upstream fix requested in https://github.com/open-telemetry/opentelemetry-ruby/issues/2497;
|
|
11
|
+
# once it ships, this subclass can be replaced by the SDK's own force_flush.
|
|
12
|
+
#
|
|
13
|
+
# @api private
|
|
14
|
+
class BatchSpanProcessor < OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor
|
|
15
|
+
# Each call captures a prefix of the FIFO queue and the current export, if any.
|
|
16
|
+
# Counts shrink on dequeue or overflow; later spans never extend a call's work.
|
|
17
|
+
#
|
|
18
|
+
# @api private
|
|
19
|
+
FlushWaiter = Struct.new(:queued, :exporting, :result) do
|
|
20
|
+
def pending?
|
|
21
|
+
queued.positive? || exporting
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Shutdown needs the upstream synchronous drain after stopping the worker.
|
|
26
|
+
# Public flush calls must never take that path, which ignores in-flight batches.
|
|
27
|
+
alias flush_at_shutdown force_flush
|
|
28
|
+
private :flush_at_shutdown
|
|
29
|
+
|
|
30
|
+
def force_flush(timeout: nil)
|
|
31
|
+
start_time = OpenTelemetry::Common::Utilities.timeout_timestamp
|
|
32
|
+
startup_error = nil
|
|
33
|
+
|
|
34
|
+
result = lock do
|
|
35
|
+
return OpenTelemetry::SDK::Trace::Export::FAILURE unless @keep_running
|
|
36
|
+
|
|
37
|
+
remaining = OpenTelemetry::Common::Utilities.maybe_timeout(timeout, start_time)
|
|
38
|
+
if remaining&.zero?
|
|
39
|
+
return OpenTelemetry::SDK::Trace::Export::SUCCESS if spans.empty? && !@exporting
|
|
40
|
+
|
|
41
|
+
return OpenTelemetry::SDK::Trace::Export::TIMEOUT
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
reset_on_fork(restart_thread: false)
|
|
45
|
+
begin
|
|
46
|
+
start_worker
|
|
47
|
+
rescue ThreadError => e
|
|
48
|
+
startup_error = e
|
|
49
|
+
next OpenTelemetry::SDK::Trace::Export::FAILURE
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
wait_for_exports(timeout, start_time)
|
|
53
|
+
end
|
|
54
|
+
# Reporting can block; release the queue lock before invoking either callback.
|
|
55
|
+
report_startup_error(startup_error) if startup_error
|
|
56
|
+
result
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# Preserve the SDK's teardown, but wake waiting flush callers and use the
|
|
60
|
+
# private upstream drain instead of dispatching back to public force_flush.
|
|
61
|
+
def shutdown(timeout: nil)
|
|
62
|
+
start_time = OpenTelemetry::Common::Utilities.timeout_timestamp
|
|
63
|
+
thread = lock do
|
|
64
|
+
return OpenTelemetry::SDK::Trace::Export::FAILURE unless @keep_running
|
|
65
|
+
|
|
66
|
+
@keep_running = false
|
|
67
|
+
@condition.signal
|
|
68
|
+
@flush_condition.broadcast
|
|
69
|
+
@thread
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
thread&.join(timeout)
|
|
73
|
+
flush_at_shutdown(timeout: OpenTelemetry::Common::Utilities.maybe_timeout(timeout, start_time))
|
|
74
|
+
dropped_spans = lock { spans.shift(spans.length) }
|
|
75
|
+
report_dropped_spans(dropped_spans, reason: "terminating") if dropped_spans.any?
|
|
76
|
+
@exporter.shutdown(timeout: OpenTelemetry::Common::Utilities.maybe_timeout(timeout, start_time))
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
private
|
|
80
|
+
|
|
81
|
+
def start_worker
|
|
82
|
+
return @thread if @thread&.alive?
|
|
83
|
+
|
|
84
|
+
@thread = Thread.new { work }
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
def report_startup_error(error)
|
|
88
|
+
@metrics_reporter.add_to_counter("otel.bsp.error", labels: { "reason" => "ThreadError" })
|
|
89
|
+
OpenTelemetry.handle_error(exception: error, message: "unexpected error in BatchSpanProcessor#start_worker")
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
def wait_for_exports(timeout, start_time)
|
|
93
|
+
waiter = FlushWaiter.new(spans.size, @exporting, OpenTelemetry::SDK::Trace::Export::SUCCESS)
|
|
94
|
+
@flush_waiters.add(waiter)
|
|
95
|
+
@condition.signal
|
|
96
|
+
|
|
97
|
+
loop do
|
|
98
|
+
return waiter.result unless waiter.pending?
|
|
99
|
+
return OpenTelemetry::SDK::Trace::Export::FAILURE unless @keep_running
|
|
100
|
+
|
|
101
|
+
remaining = OpenTelemetry::Common::Utilities.maybe_timeout(timeout, start_time)
|
|
102
|
+
return OpenTelemetry::SDK::Trace::Export::TIMEOUT if remaining&.zero?
|
|
103
|
+
|
|
104
|
+
@flush_condition.wait(@mutex, remaining)
|
|
105
|
+
end
|
|
106
|
+
ensure
|
|
107
|
+
@flush_waiters.delete(waiter)
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
# Find the batch limit while collecting waiters; Set iteration order is unspecified.
|
|
111
|
+
def queued_flushes
|
|
112
|
+
waiters = []
|
|
113
|
+
limit = batch_size
|
|
114
|
+
@flush_waiters.each do |waiter|
|
|
115
|
+
next unless waiter.queued.positive?
|
|
116
|
+
|
|
117
|
+
waiters << waiter
|
|
118
|
+
limit = waiter.queued if waiter.queued < limit
|
|
119
|
+
end
|
|
120
|
+
[waiters, limit]
|
|
121
|
+
end
|
|
122
|
+
|
|
123
|
+
# Unlike the upstream worker, a waiting flush bypasses the scheduled delay for
|
|
124
|
+
# its remaining batches. @exporting covers dequeue through result reporting.
|
|
125
|
+
def work
|
|
126
|
+
loop do
|
|
127
|
+
batch = lock do
|
|
128
|
+
waiters, limit = queued_flushes
|
|
129
|
+
if spans.size < batch_size && @keep_running && waiters.empty?
|
|
130
|
+
@condition.wait(@mutex, @delay_seconds)
|
|
131
|
+
@condition.wait(@mutex, @delay_seconds) while spans.empty? && @keep_running
|
|
132
|
+
# Waiting releases the lock; callers may register or time out before we resume.
|
|
133
|
+
waiters, limit = queued_flushes
|
|
134
|
+
end
|
|
135
|
+
return unless @keep_running
|
|
136
|
+
|
|
137
|
+
take_batch(waiters, limit)
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
export_and_notify(batch)
|
|
141
|
+
end
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
# Split at the earliest flush boundary so a batch never mixes that call's
|
|
145
|
+
# captured observations with later work. This runs under the queue lock.
|
|
146
|
+
def take_batch(waiters, limit)
|
|
147
|
+
batch = spans.shift(limit)
|
|
148
|
+
waiters.each do |waiter|
|
|
149
|
+
waiter.queued -= batch.size
|
|
150
|
+
waiter.exporting = true
|
|
151
|
+
end
|
|
152
|
+
@exporting = true
|
|
153
|
+
batch
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
def export_and_notify(batch)
|
|
157
|
+
result = OpenTelemetry::SDK::Trace::Export::FAILURE
|
|
158
|
+
@metrics_reporter.observe_value("otel.bsp.buffer_utilization", value: spans.size / max_queue_size.to_f)
|
|
159
|
+
result = export_batch(batch)
|
|
160
|
+
ensure
|
|
161
|
+
lock do
|
|
162
|
+
@flush_waiters.each do |waiter|
|
|
163
|
+
next unless waiter.exporting
|
|
164
|
+
|
|
165
|
+
unless result == OpenTelemetry::SDK::Trace::Export::SUCCESS
|
|
166
|
+
waiter.result = OpenTelemetry::SDK::Trace::Export::FAILURE
|
|
167
|
+
end
|
|
168
|
+
waiter.exporting = false
|
|
169
|
+
end
|
|
170
|
+
@exporting = false
|
|
171
|
+
@flush_condition.broadcast
|
|
172
|
+
end
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
# Queue overflow is reported inside the queue lock. Export failures are
|
|
176
|
+
# reported outside it, and are delivered to waiters by export_and_notify.
|
|
177
|
+
def report_dropped_spans(dropped_spans, reason:, **kwargs)
|
|
178
|
+
if reason == "buffer-full" && dropped_spans.any?
|
|
179
|
+
@flush_waiters.each do |waiter|
|
|
180
|
+
next unless waiter.queued.positive?
|
|
181
|
+
|
|
182
|
+
waiter.queued = [waiter.queued - dropped_spans.size, 0].max
|
|
183
|
+
waiter.result = OpenTelemetry::SDK::Trace::Export::FAILURE
|
|
184
|
+
end
|
|
185
|
+
@flush_condition.broadcast
|
|
186
|
+
end
|
|
187
|
+
super
|
|
188
|
+
end
|
|
189
|
+
|
|
190
|
+
def reset_on_fork(restart_thread: true)
|
|
191
|
+
return if @pid == Process.pid
|
|
192
|
+
|
|
193
|
+
@flush_condition = ConditionVariable.new
|
|
194
|
+
@flush_waiters = Set.new.compare_by_identity
|
|
195
|
+
@exporting = false
|
|
196
|
+
super
|
|
197
|
+
end
|
|
198
|
+
end
|
|
199
|
+
end
|
|
200
|
+
end
|
|
@@ -18,6 +18,29 @@ module NitroIntelligence
|
|
|
18
18
|
@observability_client = observability_client
|
|
19
19
|
end
|
|
20
20
|
|
|
21
|
+
# Wait for this project's observation exports queued or in flight when called.
|
|
22
|
+
#
|
|
23
|
+
# Observations completed later are excluded from this call's wait and result. The client
|
|
24
|
+
# remains usable, and active observations are not ended. Queued scores and evaluator
|
|
25
|
+
# results are not covered. Success does not guarantee immediate visibility in Cerebro.
|
|
26
|
+
#
|
|
27
|
+
# Earlier export failures are not reported, and discarded observations are not retried.
|
|
28
|
+
# Concurrent calls have independent timeouts. Completed results take precedence over
|
|
29
|
+
# timeout or shutdown.
|
|
30
|
+
#
|
|
31
|
+
# @param timeout [Numeric] maximum wait in seconds (default: 30). Exports continue after
|
|
32
|
+
# timeout. With zero, succeeds only for a running client with no pending exports.
|
|
33
|
+
# @return [Boolean] true if this call's exports succeed or none are pending; false if
|
|
34
|
+
# exporting fails or cannot start, an observation being flushed is dropped, the wait
|
|
35
|
+
# times out, or shutdown interrupts pending work. Calls made while the client is
|
|
36
|
+
# stopping or stopped return false.
|
|
37
|
+
# @example Export before submitting observation IDs for review
|
|
38
|
+
# client = NitroIntelligence.project_client_registry.fetch("my-feature-project")
|
|
39
|
+
# raise "Observation export failed or timed out" unless client.flush(timeout: 30)
|
|
40
|
+
def flush(timeout: 30) # rubocop:disable Naming/PredicateMethod
|
|
41
|
+
@observability_client.flush(timeout:) == OpenTelemetry::SDK::Trace::Export::SUCCESS
|
|
42
|
+
end
|
|
43
|
+
|
|
21
44
|
# @api private
|
|
22
45
|
def shutdown
|
|
23
46
|
@observability_client.shutdown
|
|
@@ -17,15 +17,17 @@ module NitroIntelligence
|
|
|
17
17
|
def initialize(base_url:)
|
|
18
18
|
@base_url = base_url
|
|
19
19
|
@project_clients = {}
|
|
20
|
+
@mutex = Mutex.new
|
|
20
21
|
end
|
|
21
22
|
|
|
22
23
|
# The client for the project configured under `slug`, built on first use.
|
|
24
|
+
# Concurrent fetches for the same project share one client.
|
|
23
25
|
#
|
|
24
26
|
# @param slug [String]
|
|
25
27
|
# @return [ProjectClient]
|
|
26
28
|
# @raise [Project::NotFoundError] when `observability_projects` does not configure the project
|
|
27
29
|
def fetch(slug)
|
|
28
|
-
@project_clients[slug] ||= build_project_client(slug)
|
|
30
|
+
@mutex.synchronize { @project_clients[slug] ||= build_project_client(slug) }
|
|
29
31
|
end
|
|
30
32
|
|
|
31
33
|
# Sends what each client still holds to Cerebro, and stops it. Call it as the process exits, or
|
|
@@ -33,7 +35,8 @@ module NitroIntelligence
|
|
|
33
35
|
#
|
|
34
36
|
# @return [void]
|
|
35
37
|
def shutdown_all
|
|
36
|
-
@project_clients.values
|
|
38
|
+
clients = @mutex.synchronize { @project_clients.values }
|
|
39
|
+
clients.each(&:shutdown)
|
|
37
40
|
end
|
|
38
41
|
|
|
39
42
|
private
|
|
@@ -50,6 +53,7 @@ module NitroIntelligence
|
|
|
50
53
|
config.public_key = project.public_key
|
|
51
54
|
config.secret_key = project.secret_key
|
|
52
55
|
config.base_url = @base_url
|
|
56
|
+
config.release = NitroIntelligence.configuration.current_revision
|
|
53
57
|
# Default flush of 60 seconds can be too quick when
|
|
54
58
|
# dealing with longer responses like image gen
|
|
55
59
|
config.flush_interval = 120
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
module NitroIntelligence
|
|
2
|
+
# A step of application code recorded in Cerebro, as {Tracer#trace} and {#observe} yield it.
|
|
3
|
+
#
|
|
4
|
+
# A project that masks content records the input and output given here as `[masked]`, as it does
|
|
5
|
+
# for the gem's own requests. Metadata is recorded as given, so keep content out of it.
|
|
6
|
+
class Observation
|
|
7
|
+
# The Langfuse observation types {#observe} accepts.
|
|
8
|
+
TYPES = (Langfuse::OBSERVATION_TYPES.keys - [:event]).freeze
|
|
9
|
+
|
|
10
|
+
# @api private
|
|
11
|
+
MAX_STATUS_MESSAGE_LENGTH = 2000
|
|
12
|
+
|
|
13
|
+
# @api private
|
|
14
|
+
def initialize(langfuse_observation, observability_client:, project:)
|
|
15
|
+
@observation = langfuse_observation
|
|
16
|
+
@observability_client = observability_client
|
|
17
|
+
@project = project
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
# @return [String] the observation's ID, for scoring it with {Reporter#score}
|
|
21
|
+
def id
|
|
22
|
+
@observation.id
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# @return [String] the ID of the trace the observation is in, for scoring it with {Reporter#score}
|
|
26
|
+
def trace_id
|
|
27
|
+
@observation.trace_id
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
# Records what the step was given. On the observation {Tracer#trace} yields, this is the
|
|
31
|
+
# trace's input too.
|
|
32
|
+
#
|
|
33
|
+
# @param value [Object] anything that serializes to JSON
|
|
34
|
+
def input=(value)
|
|
35
|
+
@observation.input = @project.mask(value)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
# Records what the step produced. On the observation {Tracer#trace} yields, this is the
|
|
39
|
+
# trace's output too.
|
|
40
|
+
#
|
|
41
|
+
# @param value [Object] anything that serializes to JSON
|
|
42
|
+
def output=(value)
|
|
43
|
+
@observation.output = @project.mask(value)
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
# Records metadata on the observation. It is not masked.
|
|
47
|
+
#
|
|
48
|
+
# @param value [Hash]
|
|
49
|
+
def metadata=(value)
|
|
50
|
+
@observation.metadata = value
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# Records a step inside this one, around a block. The gem's requests made inside the block nest
|
|
54
|
+
# under the step. An exception the block raises is recorded on the step at level `ERROR`, and
|
|
55
|
+
# re-raised.
|
|
56
|
+
#
|
|
57
|
+
# @param name [String] what the step does
|
|
58
|
+
# @param as_type [Symbol] one of {TYPES}, such as `:span`, `:retriever` or `:tool`
|
|
59
|
+
# @param input [Object, nil] what the step is given
|
|
60
|
+
# @param metadata [Hash, nil]
|
|
61
|
+
# @yieldparam observation [Observation] the step, to record its output on
|
|
62
|
+
# @return [Object] the block's value
|
|
63
|
+
# @raise [ArgumentError] when `as_type` is not one of {TYPES}, or no block is given
|
|
64
|
+
# @example
|
|
65
|
+
# hits = root_observation.observe("Elasticsearch", as_type: :retriever, input: { k: 50 }) do |step|
|
|
66
|
+
# search(question).tap { |hits| step.output = hits.map(&:page_id) }
|
|
67
|
+
# end
|
|
68
|
+
def observe(name, as_type: :span, input: nil, metadata: nil, &)
|
|
69
|
+
raise ArgumentError, "a block is required" unless block_given?
|
|
70
|
+
unless TYPES.include?(as_type)
|
|
71
|
+
raise ArgumentError, "as_type must be one of #{TYPES.join(', ')}, not #{as_type.inspect}"
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
child = @observability_client.start_observation(
|
|
75
|
+
name,
|
|
76
|
+
attributes(input:, metadata:),
|
|
77
|
+
as_type:,
|
|
78
|
+
parent_span_context: @observation.otel_span.context
|
|
79
|
+
)
|
|
80
|
+
Observation.new(child, observability_client: @observability_client, project: @project).run(&)
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
# Records something that happened at a point in time inside this step, such as a user's choice.
|
|
84
|
+
#
|
|
85
|
+
# @param name [String] what happened
|
|
86
|
+
# @param input [Object, nil]
|
|
87
|
+
# @param output [Object, nil]
|
|
88
|
+
# @param metadata [Hash, nil]
|
|
89
|
+
# @return [void]
|
|
90
|
+
def event(name, input: nil, output: nil, metadata: nil)
|
|
91
|
+
@observability_client.start_observation(
|
|
92
|
+
name,
|
|
93
|
+
attributes(input:, output:, metadata:),
|
|
94
|
+
as_type: :event,
|
|
95
|
+
parent_span_context: @observation.otel_span.context
|
|
96
|
+
)
|
|
97
|
+
nil
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
# @api private
|
|
101
|
+
#
|
|
102
|
+
# Runs the block with this observation active, records an exception it raises, and ends the
|
|
103
|
+
# observation.
|
|
104
|
+
def run
|
|
105
|
+
@observability_client.within(@observation) { yield self }
|
|
106
|
+
rescue => e
|
|
107
|
+
@observation.update(level: "ERROR", status_message: failure_message(e))
|
|
108
|
+
raise
|
|
109
|
+
ensure
|
|
110
|
+
@observation.end
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
private
|
|
114
|
+
|
|
115
|
+
def attributes(input: nil, output: nil, metadata: nil)
|
|
116
|
+
{
|
|
117
|
+
input: @project.mask(input),
|
|
118
|
+
output: @project.mask(output),
|
|
119
|
+
metadata:,
|
|
120
|
+
environment: NitroIntelligence.environment.to_s,
|
|
121
|
+
}.compact
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
# The message of an exception can quote what the step was given, so a masked project records
|
|
125
|
+
# which exception it was and nothing it said.
|
|
126
|
+
def failure_message(error)
|
|
127
|
+
return error.class.to_s if @project.mask_content?
|
|
128
|
+
|
|
129
|
+
"#{error.class}: #{error.message}".truncate(MAX_STATUS_MESSAGE_LENGTH)
|
|
130
|
+
end
|
|
131
|
+
end
|
|
132
|
+
end
|
|
@@ -53,24 +53,36 @@ module NitroIntelligence
|
|
|
53
53
|
response
|
|
54
54
|
end
|
|
55
55
|
|
|
56
|
-
# Scores a trace. Scoring it again under the same `id` replaces the
|
|
56
|
+
# Scores a trace, or an observation in it. Scoring it again under the same `id` replaces the
|
|
57
|
+
# score.
|
|
57
58
|
#
|
|
58
|
-
# @param trace_id [String] the trace, such as {Trace.create_id} gives for a seed
|
|
59
|
+
# @param trace_id [String] the trace, such as {Trace.create_id} gives for a seed, or
|
|
60
|
+
# {Observation#trace_id}
|
|
59
61
|
# @param name [String] what the score measures
|
|
60
62
|
# @param value [Numeric, String, Boolean] the score: a number, `true` or `false` for `:boolean`,
|
|
61
63
|
# or the category's name for `:categorical`
|
|
64
|
+
# @param observation_id [String, nil] the observation in the trace to score, as
|
|
65
|
+
# {Observation#id} gives it, rather than the trace as a whole
|
|
62
66
|
# @param id [String] identifies the score, so that recording it again replaces it. Defaults to
|
|
63
|
-
# one per trace and name.
|
|
67
|
+
# one per trace, observation and name.
|
|
64
68
|
# @param data_type [Symbol] `:numeric`, `:boolean` or `:categorical`
|
|
69
|
+
# @param comment [String, nil]
|
|
70
|
+
# @param metadata [Hash, nil]
|
|
65
71
|
# @return [void]
|
|
66
72
|
# @raise [ArgumentError] when `value` does not suit `data_type`
|
|
67
|
-
|
|
73
|
+
# @example Scoring a search result the user clicked, from the request that records the click
|
|
74
|
+
# reporter.score(trace_id: params[:trace_id], name: "clicked", value: 1, metadata: { position: 3 })
|
|
75
|
+
def score(trace_id:, name:, value:, observation_id: nil, id: [trace_id, observation_id, name].compact.join("-"), # rubocop:disable Metrics/ParameterLists
|
|
76
|
+
data_type: :numeric, comment: nil, metadata: nil)
|
|
68
77
|
@project_client.observability_client.create_score(
|
|
69
78
|
id:,
|
|
70
79
|
trace_id:,
|
|
80
|
+
observation_id:,
|
|
71
81
|
name:,
|
|
72
82
|
value:,
|
|
73
83
|
data_type:,
|
|
84
|
+
comment:,
|
|
85
|
+
metadata:,
|
|
74
86
|
environment: NitroIntelligence.environment
|
|
75
87
|
)
|
|
76
88
|
end
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
require "nitro_intelligence/observation"
|
|
2
|
+
require "nitro_intelligence/trace"
|
|
3
|
+
|
|
4
|
+
module NitroIntelligence
|
|
5
|
+
# Records application code in Cerebro as a trace, with the gem's requests made inside it nested
|
|
6
|
+
# under it, for a feature whose work is more than one model call.
|
|
7
|
+
#
|
|
8
|
+
# @example
|
|
9
|
+
# tracer = NitroIntelligence::Tracer.new(observability_project_slug: "page-finder")
|
|
10
|
+
# client = NitroIntelligence::Client.new(observability_project_slug: "page-finder")
|
|
11
|
+
#
|
|
12
|
+
# tracer.trace("Page Finder search", input: { question: }, user_id: current_user.id.to_s) do |root_observation|
|
|
13
|
+
# query = client.chat(message: question) # recorded inside the trace
|
|
14
|
+
# hits = root_observation.observe("Elasticsearch", as_type: :retriever, input: { k: 50 }) do |step|
|
|
15
|
+
# search(query).tap { |hits| step.output = hits.map(&:page_id) }
|
|
16
|
+
# end
|
|
17
|
+
# root_observation.output = hits.map(&:page_id)
|
|
18
|
+
# end
|
|
19
|
+
class Tracer
|
|
20
|
+
# @param observability_project_slug [String] the Cerebro project to record in. The gem's
|
|
21
|
+
# requests nest under the trace only when their client records in the same project.
|
|
22
|
+
# @raise [Observability::Project::NotFoundError] when `observability_projects` does not configure
|
|
23
|
+
# the project
|
|
24
|
+
def initialize(observability_project_slug:)
|
|
25
|
+
@project_client = NitroIntelligence.project_client_registry.fetch(observability_project_slug)
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# Records the block as a trace of its own, even inside another trace, with its environment and
|
|
29
|
+
# release set as the gem's requests set them. The gem's requests made inside the block, through a
|
|
30
|
+
# client for the same project and without a `trace_seed`, are recorded as observations in this
|
|
31
|
+
# trace instead of traces of their own: the trace's name, input, output, user, session and tags are
|
|
32
|
+
# the ones given here, and the request's `trace_name`, `user_id`, `session_id` and `tags` are not
|
|
33
|
+
# recorded. An exception the block raises is recorded on the trace at level `ERROR`, and re-raised.
|
|
34
|
+
#
|
|
35
|
+
# @param name [String] the trace's name
|
|
36
|
+
# @param as_type [Symbol] the type of the trace's root observation, one of {Observation::TYPES}
|
|
37
|
+
# @param input [Object, nil] the trace's input. Set it later with {Observation#input=}.
|
|
38
|
+
# @param metadata [Hash] recorded on the trace and every observation in it, with each value
|
|
39
|
+
# converted to a String. It is not masked.
|
|
40
|
+
# @param user_id [String, nil] the application's user, defaulting to `observability_user_id`
|
|
41
|
+
# @param session_id [String, nil] groups related traces
|
|
42
|
+
# @param tags [Array<String>, nil] labels the trace for filtering
|
|
43
|
+
# @param trace_seed [String, nil] gives the trace the ID {Trace.create_id} gives for the seed,
|
|
44
|
+
# rather than a random one
|
|
45
|
+
# @yieldparam root_observation [Observation] the trace's root observation
|
|
46
|
+
# @return [Object] the block's value
|
|
47
|
+
# @raise [ArgumentError] when `as_type` is not one of {Observation::TYPES}, or no block is given
|
|
48
|
+
def trace(name, as_type: :span, input: nil, metadata: {}, user_id: nil, session_id: nil, tags: nil, # rubocop:disable Metrics/ParameterLists
|
|
49
|
+
trace_seed: nil, &)
|
|
50
|
+
raise ArgumentError, "a block is required" unless block_given?
|
|
51
|
+
unless Observation::TYPES.include?(as_type)
|
|
52
|
+
raise ArgumentError, "as_type must be one of #{Observation::TYPES.join(', ')}, not #{as_type.inspect}"
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
propagated = propagated_attributes(metadata:, user_id:, session_id:, tags:)
|
|
56
|
+
LangfuseExtension.in_new_trace do
|
|
57
|
+
Langfuse.propagate_attributes(**propagated) { start(name, as_type:, input:, trace_seed:).run(&) }
|
|
58
|
+
end
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
private
|
|
62
|
+
|
|
63
|
+
def start(name, as_type:, input:, trace_seed:)
|
|
64
|
+
observability_client = @project_client.observability_client
|
|
65
|
+
trace_id = trace_seed.present? ? Trace.create_id(seed: trace_seed) : Langfuse.create_trace_id
|
|
66
|
+
|
|
67
|
+
langfuse_observation = observability_client.start_observation(
|
|
68
|
+
name, { environment: NitroIntelligence.environment.to_s }, as_type:, trace_id:
|
|
69
|
+
)
|
|
70
|
+
langfuse_observation.update_trace(name:)
|
|
71
|
+
|
|
72
|
+
Observation.new(
|
|
73
|
+
langfuse_observation, observability_client:, project: @project_client.project
|
|
74
|
+
).tap { |observation| observation.input = input unless input.nil? }
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
def propagated_attributes(metadata:, user_id:, session_id:, tags:)
|
|
78
|
+
attributes = {
|
|
79
|
+
user_id: user_id || NitroIntelligence.configuration.observability_user_id,
|
|
80
|
+
metadata: metadata.to_h.transform_values(&:to_s),
|
|
81
|
+
}
|
|
82
|
+
attributes[:session_id] = session_id if session_id.present?
|
|
83
|
+
attributes[:tags] = tags if tags.present?
|
|
84
|
+
attributes
|
|
85
|
+
end
|
|
86
|
+
end
|
|
87
|
+
end
|
data/lib/nitro_intelligence.rb
CHANGED
|
@@ -16,6 +16,7 @@ require "nitro_intelligence/media/image_generation"
|
|
|
16
16
|
require "nitro_intelligence/models/model_catalog"
|
|
17
17
|
require "nitro_intelligence/observability/project_client_registry"
|
|
18
18
|
require "nitro_intelligence/reporter"
|
|
19
|
+
require "nitro_intelligence/tracer"
|
|
19
20
|
|
|
20
21
|
# The Ruby client for the Nitro Intelligence Platform.
|
|
21
22
|
#
|
|
@@ -28,6 +29,8 @@ module NitroIntelligence
|
|
|
28
29
|
# @return [Class<Configuration>]
|
|
29
30
|
mattr_accessor :configuration, default: Configuration
|
|
30
31
|
|
|
32
|
+
@project_client_registry_mutex = Mutex.new
|
|
33
|
+
|
|
31
34
|
class << self
|
|
32
35
|
# @!method configure
|
|
33
36
|
# Sets the configuration, normally once, from an initializer.
|
|
@@ -76,13 +79,15 @@ module NitroIntelligence
|
|
|
76
79
|
@model_catalog ||= ModelCatalog.new(configuration.model_config)
|
|
77
80
|
end
|
|
78
81
|
|
|
79
|
-
# Each Cerebro project's client.
|
|
82
|
+
# Each Cerebro project's client, through a registry shared by all threads.
|
|
80
83
|
#
|
|
81
84
|
# @return [Observability::ProjectClientRegistry]
|
|
82
85
|
def project_client_registry
|
|
83
|
-
@
|
|
84
|
-
|
|
85
|
-
|
|
86
|
+
@project_client_registry_mutex.synchronize do
|
|
87
|
+
@project_client_registry ||= Observability::ProjectClientRegistry.new(
|
|
88
|
+
base_url: configuration.observability_base_url
|
|
89
|
+
)
|
|
90
|
+
end
|
|
86
91
|
end
|
|
87
92
|
end
|
|
88
93
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: nitro_intelligence
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 4.
|
|
4
|
+
version: 4.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Igor Artemenko
|
|
@@ -65,6 +65,26 @@ dependencies:
|
|
|
65
65
|
- - "~>"
|
|
66
66
|
- !ruby/object:Gem::Version
|
|
67
67
|
version: '0.86'
|
|
68
|
+
- !ruby/object:Gem::Dependency
|
|
69
|
+
name: opentelemetry-sdk
|
|
70
|
+
requirement: !ruby/object:Gem::Requirement
|
|
71
|
+
requirements:
|
|
72
|
+
- - ">="
|
|
73
|
+
- !ruby/object:Gem::Version
|
|
74
|
+
version: '1.10'
|
|
75
|
+
- - "<"
|
|
76
|
+
- !ruby/object:Gem::Version
|
|
77
|
+
version: '1.14'
|
|
78
|
+
type: :runtime
|
|
79
|
+
prerelease: false
|
|
80
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
81
|
+
requirements:
|
|
82
|
+
- - ">="
|
|
83
|
+
- !ruby/object:Gem::Version
|
|
84
|
+
version: '1.10'
|
|
85
|
+
- - "<"
|
|
86
|
+
- !ruby/object:Gem::Version
|
|
87
|
+
version: '1.14'
|
|
68
88
|
description: The Ruby client for Nitro Intelligence
|
|
69
89
|
email:
|
|
70
90
|
- igor.artemenko@powerhrg.com
|
|
@@ -111,6 +131,7 @@ files:
|
|
|
111
131
|
- lib/nitro_intelligence/models/model_catalog.rb
|
|
112
132
|
- lib/nitro_intelligence/models/model_factory.rb
|
|
113
133
|
- lib/nitro_intelligence/null_cache.rb
|
|
134
|
+
- lib/nitro_intelligence/observability/batch_span_processor.rb
|
|
114
135
|
- lib/nitro_intelligence/observability/project.rb
|
|
115
136
|
- lib/nitro_intelligence/observability/project_client.rb
|
|
116
137
|
- lib/nitro_intelligence/observability/project_client_registry.rb
|
|
@@ -118,6 +139,7 @@ files:
|
|
|
118
139
|
- lib/nitro_intelligence/observability/prompt_resolver.rb
|
|
119
140
|
- lib/nitro_intelligence/observability/prompt_store.rb
|
|
120
141
|
- lib/nitro_intelligence/observability/upload_handler.rb
|
|
142
|
+
- lib/nitro_intelligence/observation.rb
|
|
121
143
|
- lib/nitro_intelligence/pending_review.rb
|
|
122
144
|
- lib/nitro_intelligence/railtie.rb
|
|
123
145
|
- lib/nitro_intelligence/reporter.rb
|
|
@@ -126,6 +148,7 @@ files:
|
|
|
126
148
|
- lib/nitro_intelligence/tool_call_review_interrupt.rb
|
|
127
149
|
- lib/nitro_intelligence/tool_call_review_validator.rb
|
|
128
150
|
- lib/nitro_intelligence/trace.rb
|
|
151
|
+
- lib/nitro_intelligence/tracer.rb
|
|
129
152
|
- lib/nitro_intelligence/version.rb
|
|
130
153
|
homepage: https://github.com/powerhome/nitro-intelligence.rb
|
|
131
154
|
licenses:
|