chronos-ruby 0.6.0.pre.1 → 0.8.0.pre.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +48 -0
- data/README.md +53 -10
- data/contracts/apm-batch-v1.schema.json +35 -0
- data/contracts/dependencies-v1.schema.json +38 -0
- data/contracts/event-v1.schema.json +1 -1
- data/docs/adr/ADR-015-bounded-apm-aggregation.md +27 -0
- data/docs/adr/ADR-016-explicit-observability-integrations.md +25 -0
- data/docs/architecture.md +11 -2
- data/docs/compatibility.md +4 -0
- data/docs/configuration.md +28 -1
- data/docs/data-collected.md +17 -3
- data/docs/examples/plain-ruby.md +7 -1
- data/docs/modules/apm-aggregation.md +57 -0
- data/docs/modules/cache-observability.md +16 -0
- data/docs/modules/configuration.md +4 -0
- data/docs/modules/dependencies.md +18 -0
- data/docs/modules/external-http.md +27 -0
- data/docs/modules/rails-legacy.md +2 -2
- data/docs/modules/sidekiq-legacy.md +2 -2
- data/docs/modules/telemetry-events.md +2 -2
- data/docs/performance.md +29 -1
- data/docs/privacy-lgpd.md +16 -2
- data/docs/troubleshooting.md +25 -1
- data/lib/chronos/agent.rb +49 -0
- data/lib/chronos/application/apm_aggregator.rb +270 -0
- data/lib/chronos/application/apm_error_classifier.rb +27 -0
- data/lib/chronos/application/capture_telemetry.rb +45 -5
- data/lib/chronos/application/delivery_pipeline.rb +7 -0
- data/lib/chronos/application/dependency_reporter.rb +129 -0
- data/lib/chronos/application/remote_configuration.rb +8 -1
- data/lib/chronos/configuration/apm_validation.rb +76 -0
- data/lib/chronos/configuration.rb +35 -2
- data/lib/chronos/core/cache_normalizer.rb +99 -0
- data/lib/chronos/core/metric_aggregate.rb +113 -0
- data/lib/chronos/core/sql_normalizer.rb +114 -0
- data/lib/chronos/core/telemetry_event.rb +1 -1
- data/lib/chronos/integrations/net_http.rb +165 -0
- data/lib/chronos/integrations/rack/middleware.rb +19 -1
- data/lib/chronos/integrations/sidekiq.rb +2 -0
- data/lib/chronos/net_http.rb +4 -0
- data/lib/chronos/observability_facade.rb +43 -0
- data/lib/chronos/rails/notifications_subscriber.rb +42 -9
- data/lib/chronos/version.rb +1 -1
- data/lib/chronos.rb +23 -0
- metadata +19 -1
|
@@ -29,7 +29,7 @@ The public `ActiveSupport::Notifications.subscribe` API is used for:
|
|
|
29
29
|
| `sql.active_record` | `query` | operation name, cached flag, and duration |
|
|
30
30
|
| `deliver.action_mailer` | `job` with `kind=mailer` | mailer, action, and duration |
|
|
31
31
|
| `perform.active_job` | `job` | job class, queue, and duration, when Active Job is available |
|
|
32
|
-
| cache read/write/hit notifications | `cache` | operation,
|
|
32
|
+
| cache read/write/hit notifications | `cache` | operation, backend, namespace, hit/miss, duration, and optional scoped key hash |
|
|
33
33
|
|
|
34
34
|
Raw SQL, binds, cache keys, mail recipients and bodies, job IDs and arguments, request/response bodies, cookies, and authorization headers are not copied. Template identifiers are reduced to their basename. Every event passes through `Sanitizer`, `SafeSerializer`, the bounded queue, retry policy, circuit breaker, and memory backlog.
|
|
35
35
|
|
|
@@ -57,4 +57,4 @@ The repository contains independent applications under `examples/rails-4.2` and
|
|
|
57
57
|
|
|
58
58
|
## Limits
|
|
59
59
|
|
|
60
|
-
Version 0.
|
|
60
|
+
Version 0.7 aggregates controller, SQL, and job timings into bounded APM batches. SQL literals and binds remain excluded while a bounded normalized query and fingerprint are produced. Sidekiq support remains a separate optional integration; full Active Job propagation remains a later legacy increment.
|
|
@@ -4,7 +4,7 @@ Version `0.6.0.pre.1` starts the legacy jobs line with optional Sidekiq 4 and 5
|
|
|
4
4
|
|
|
5
5
|
```ruby
|
|
6
6
|
gem "sidekiq", "~> 5.0"
|
|
7
|
-
gem "chronos-ruby", "0.
|
|
7
|
+
gem "chronos-ruby", "0.8.0.pre.1", :require => "chronos/sidekiq"
|
|
8
8
|
```
|
|
9
9
|
|
|
10
10
|
`chronos/sidekiq` installs middleware through the public `configure_client` and `configure_server` APIs. It does nothing when Sidekiq is unavailable, and the core gem never requires Sidekiq. Installation adds no Chronos thread or Redis/database connection per job; delivery continues through the agent's existing fixed worker pool.
|
|
@@ -13,7 +13,7 @@ The client middleware adds a top-level `chronos` metadata object to the Sidekiq
|
|
|
13
13
|
|
|
14
14
|
Collected Sidekiq fields are worker class, queue, JID, retry count, duration, queue latency, status, error class, bounded tags, and bounded arguments. At most 20 top-level arguments, 20 values per nested collection, four levels, and 512 bytes per string are traversed. Arguments then pass through the standard sensitive-key/content sanitizer before queueing. Limits reduce exposure and resource use but do not make unnecessary personal data safe to collect.
|
|
15
15
|
|
|
16
|
-
On failure, the server middleware records failed-job
|
|
16
|
+
On failure, the server middleware records a failed-job observation, invokes `notify_once`, and re-raises the identical exception so Sidekiq retains ownership of retries and failure handlers. A shared execution marker prevents a nested Active Job hook and the Sidekiq middleware from sending the same exception twice. Version 0.7 aggregates the job observation; the exception retains sanitized bounded arguments. Chronos does not install an additional global Sidekiq error handler.
|
|
17
17
|
|
|
18
18
|
Known limitations for this first 0.6 prerelease:
|
|
19
19
|
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Framework telemetry events
|
|
2
2
|
|
|
3
|
-
`Chronos::Core::TelemetryEvent` and `Chronos::Core::TelemetrySerializer` extend the v1 envelope to `request`, `query`, `job`, and `
|
|
3
|
+
`Chronos::Core::TelemetryEvent` and `Chronos::Core::TelemetrySerializer` extend the v1 envelope to `request`, `query`, `job`, `cache`, `external_http`, `dependencies`, and `metric_batch` events. These values exist so integrations do not model operational telemetry as fake exceptions.
|
|
4
4
|
|
|
5
5
|
Telemetry has the same `schema_version`, event ID, timestamps, project, environment, service, runtime, context, sanitization, payload-size limit, idempotency header, asynchronous queue, retry, circuit breaker, and memory backlog used by exception events. Unsupported types are rejected locally.
|
|
6
6
|
|
|
7
7
|
`Chronos.record_event` is the narrow integration entry point. Application code should prefer documented higher-level APIs; its payload is allowlisted by each bundled integration and then sanitized. Remote configuration can reduce or disable any locally enabled telemetry type but cannot enable a type excluded by local configuration.
|
|
8
8
|
|
|
9
|
-
Version 0.
|
|
9
|
+
Version 0.8 aggregates request, query, job, and enabled external HTTP observations into `metric_batch` events by default. Cache remains an individual event while contributing to an existing traced request breakdown. Dependencies use one separate event per agent. Set `apm_enabled = false` only when individual legacy telemetry is required for diagnostics. Percentiles remain server-side; see [Essential APM aggregation](apm-aggregation.md).
|
data/docs/performance.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Performance
|
|
2
2
|
|
|
3
|
-
Performance is a functional requirement, but version 0.
|
|
3
|
+
Performance is a functional requirement, but version 0.8 makes no unverified speed claim.
|
|
4
4
|
|
|
5
5
|
Current controls:
|
|
6
6
|
|
|
@@ -20,6 +20,10 @@ Current controls:
|
|
|
20
20
|
- Rack middleware never consumes request or response bodies.
|
|
21
21
|
- Rails subscribers copy only small allowlisted field sets and never copy raw SQL or job arguments.
|
|
22
22
|
- Sidekiq middleware creates no per-job thread or connection and bounds arguments, collections, nesting, strings, and tags before telemetry capture.
|
|
23
|
+
- APM group, trace, query-fingerprint, histogram, and batch counts are fixed; no APM timer thread is created.
|
|
24
|
+
- outbound HTTP instrumentation uses two clock reads and bounded metadata without body/header traversal;
|
|
25
|
+
- cache normalization is bounded and SHA-256 runs only when explicitly enabled;
|
|
26
|
+
- dependency inventory runs at most once per agent and is capped at 200 loaded specs.
|
|
23
27
|
|
|
24
28
|
Run the scripts under `benchmarks/` and record Ruby version, operating system, CPU, warmup, iteration count, median, and dispersion before publishing results. `benchmarks/filtering.rb` measures privacy filtering, `benchmarks/retry_backlog.rb` measures fixed-memory outage behavior, `benchmarks/request_overhead.rb` compares Rack-protocol calls, and `benchmarks/rails_notifications.rb` isolates subscriber normalization overhead.
|
|
25
29
|
|
|
@@ -87,3 +91,27 @@ ITERATIONS=100000 bundle _1.17.3_ exec ruby benchmarks/sidekiq_middleware.rb
|
|
|
87
91
|
```
|
|
88
92
|
|
|
89
93
|
The benchmark executes client context injection and successful server normalization with a no-op notifier. It includes bounded argument handling and clock reads, but excludes Sidekiq/Redis, serialization, queueing, and network delivery. Record a controlled result before making a performance claim.
|
|
94
|
+
|
|
95
|
+
## Version 0.7 APM aggregation benchmark
|
|
96
|
+
|
|
97
|
+
Run:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
ITERATIONS=100000 bundle _1.17.3_ exec ruby benchmarks/apm_aggregation.rb
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
The fixture repeatedly updates one SQL metric group and one trace-local fingerprint. It excludes ActiveSupport dispatch, serialization, queueing, and network delivery. The benchmark must report one retained group; memory must remain bounded by configuration. Record runtime, hardware, warmup, median, and dispersion before publishing a performance claim.
|
|
104
|
+
|
|
105
|
+
A local diagnostic run on 2026-07-20 used Ruby 2.2.10 on macOS arm64 and 10,000 observations. It retained one group and measured approximately 34.972 microseconds per observation. This single run has no controlled warmup, median, or dispersion and is not a production performance claim.
|
|
106
|
+
|
|
107
|
+
## Version 0.8 external HTTP benchmark
|
|
108
|
+
|
|
109
|
+
Run:
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
ITERATIONS=100000 bundle _1.17.3_ exec ruby benchmarks/external_http.rb
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
The fixture compares a no-op Net::HTTP-compatible object with the per-instance wrapper. It includes trace-header injection, bounded outcome capture, and clock reads, but excludes DNS, sockets, TLS, serialization, queueing, and network delivery. Record a controlled result before making a performance claim.
|
|
116
|
+
|
|
117
|
+
A local diagnostic run on 2026-07-20 used Ruby 2.2.10 on macOS arm64 and 10,000 calls. It measured approximately 15.155 microseconds of wrapper overhead per call. This single run has no controlled warmup, median, or dispersion and is not a production performance claim.
|
data/docs/privacy-lgpd.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Privacy and LGPD
|
|
2
2
|
|
|
3
|
-
Version 0.
|
|
3
|
+
Version 0.8 sanitizes exception, framework telemetry, dependency inventory, and APM metric batches before JSON serialization, queueing, retry backlog, or transport. This reduces accidental exposure, but the host application remains responsible for lawful purpose, minimization, access control, retention, and responses to data-subject requests.
|
|
4
4
|
|
|
5
5
|
## Default policy
|
|
6
6
|
|
|
@@ -15,8 +15,18 @@ Version 0.6 sanitizes exception and framework telemetry before JSON serializatio
|
|
|
15
15
|
| Unknown Ruby objects | Represented by class name without calling application serialization |
|
|
16
16
|
| Request/response bodies, raw query strings, cookies, authorization headers, raw SQL/binds, cache values, mail bodies, environment variables | Never collected automatically |
|
|
17
17
|
| Sidekiq arguments | Collected automatically, limited before sanitization, then redacted by the common policy |
|
|
18
|
+
| SQL | Comments, quoted/numeric/boolean/null literals removed; binds never read; bounded identifiers remain |
|
|
19
|
+
| External HTTP | Host/method/status/timing only; URL path/query, Authorization, bodies, headers, and error messages omitted |
|
|
20
|
+
| Cache key | Omitted by default; optional project-scoped SHA-256 hash; cache value never read |
|
|
21
|
+
| Dependencies | Bounded loaded gem names/versions and detected runtime labels; paths and lockfiles omitted |
|
|
18
22
|
|
|
19
|
-
The retry backlog exists only in process memory, accepts only `SerializedEvent`, has a fixed capacity, and disappears on process exit. Version 0.
|
|
23
|
+
The retry backlog exists only in process memory, accepts only `SerializedEvent`, has a fixed capacity, and disappears on process exit. Version 0.8 does not persist telemetry to disk.
|
|
24
|
+
|
|
25
|
+
## External services, cache, and dependencies
|
|
26
|
+
|
|
27
|
+
Outbound HTTP instrumentation is disabled by default and installed per connection. Trace headers reveal correlation identifiers to the selected destination; enable propagation only for destinations permitted to receive them. The wrapper does not inspect any other header or URL component.
|
|
28
|
+
|
|
29
|
+
Raw cache keys can contain user or business data and are never delivered. Optional hashing is pseudonymization, not anonymization: predictable low-entropy keys can still be guessed. Dependency reporting is a separate once-per-agent event, never copied into exceptions, and can be disabled. It reads loaded gem name/version pairs only, with no filesystem path, source, configuration, or complete lockfile.
|
|
20
30
|
|
|
21
31
|
## Rails telemetry
|
|
22
32
|
|
|
@@ -26,6 +36,10 @@ Rails subscribers use per-notification allowlists. SQL events retain only the op
|
|
|
26
36
|
|
|
27
37
|
The optional Sidekiq middleware is the only version 0.6 integration that automatically reads job arguments. It traverses at most 20 top-level arguments, 20 items per nested collection, four levels, and 512 bytes per string. These structural limits run before the common key and content sanitizer. Trace propagation contains only trace and request identifiers. Do not place credentials or unnecessary personal, health, or financial data in job arguments; configure application-specific blocklist keys and audit representative synthetic payloads before enabling production delivery.
|
|
28
38
|
|
|
39
|
+
## APM dimensions
|
|
40
|
+
|
|
41
|
+
Metric groups deliberately exclude user IDs, JIDs, raw URLs, request parameters, bind values, exception messages, and cache keys. SQL normalization removes common literal forms and comments before fingerprinting, but retains bounded database identifiers and cannot parse every dialect. Do not encode personal or secret values in schema, table, column, SQL keyword, or operation names. Slow-query source contains a bounded file/line frame under the configured application root, not source-code contents.
|
|
42
|
+
|
|
29
43
|
## Rack context and breadcrumbs
|
|
30
44
|
|
|
31
45
|
The Rack middleware copies only bounded operational fields and parameter hashes that another component already parsed. It does not read `rack.input` or copy `QUERY_STRING`. User context is opt-in through `chronos.user`, and user agent collection is disabled unless the middleware option enables it.
|
data/docs/troubleshooting.md
CHANGED
|
@@ -22,7 +22,19 @@ Require `chronos/rails` from the generated initializer and confirm it runs befor
|
|
|
22
22
|
|
|
23
23
|
## Rails telemetry contains no SQL or cache key
|
|
24
24
|
|
|
25
|
-
This is intentional.
|
|
25
|
+
This is intentional. Chronos records bounded SQL and cache operational metadata without raw statements, binds, keys, or values. Set `cache_key_mode = :sha256` before configuration only when a stable pseudonymous key identity is required.
|
|
26
|
+
|
|
27
|
+
## External HTTP telemetry is missing
|
|
28
|
+
|
|
29
|
+
Set `external_http_enabled = true`, then call `Chronos.instrument_net_http(http)` for each `Net::HTTP` instance after Chronos is configured. Existing and future objects are not patched globally. The method returns `false` when disabled, already installed, incompatible, or unconfigured.
|
|
30
|
+
|
|
31
|
+
## Trace headers should not reach a destination
|
|
32
|
+
|
|
33
|
+
Set `external_http_trace_headers = false` before configuring the agent, or do not instrument that connection. Chronos preserves an application-supplied Chronos header instead of overwriting it.
|
|
34
|
+
|
|
35
|
+
## Dependency inventory is missing or incomplete
|
|
36
|
+
|
|
37
|
+
The event is emitted once per agent and contains only gems loaded at collection time, capped by `dependency_max_items`. It does not parse the lockfile or activate optional frameworks. Call `Chronos.report_dependencies` after application boot if the first event can occur before all integrations load, or set `dependency_reporting = false` to disable collection.
|
|
26
38
|
|
|
27
39
|
## Sidekiq telemetry is missing
|
|
28
40
|
|
|
@@ -32,6 +44,18 @@ Require `chronos/sidekiq` after Sidekiq is available and configure Chronos befor
|
|
|
32
44
|
|
|
33
45
|
The bundled middleware does not install a global error handler and uses `notify_once` inside a shared job scope. Check for application-installed Chronos calls or third-party global handlers outside that scope. Keep a single bundled server middleware entry in the Sidekiq chain.
|
|
34
46
|
|
|
47
|
+
## Request, query, or job events are not sent immediately
|
|
48
|
+
|
|
49
|
+
This is expected with version 0.7 APM aggregation. Groups drain after `apm_flush_count` observations or during `Chronos.flush`/`Chronos.close`. Inspect `agent.diagnostics[:apm]` when using an explicit agent. Set `apm_enabled = false` only for temporary individual-event diagnostics.
|
|
50
|
+
|
|
51
|
+
## Query metrics contain no literal values or binds
|
|
52
|
+
|
|
53
|
+
This is intentional. The normalizer removes common literal forms and never reads binds. Queries differing only by values should share a fingerprint. If an unsupported database dialect leaves a sensitive literal form, stop delivery, add a synthetic privacy fixture, and report the dialect through the security/support process.
|
|
54
|
+
|
|
55
|
+
## Possible N+1 or deadlock signal is inaccurate
|
|
56
|
+
|
|
57
|
+
Local detectors emit bounded heuristics, not confirmed diagnoses. Repetition can be legitimate and exception class names can be adapter-specific. Confirm the trace in the SaaS and application logs before changing application behavior.
|
|
58
|
+
|
|
35
59
|
## Context appears missing
|
|
36
60
|
|
|
37
61
|
The legacy context store is thread-local. A new application-created thread does not inherit context. Establish a new `Chronos.with_context` scope inside that thread, and issue manual notification before the scope exits. For Rack capture, supply user and explicit parameters through the documented environment keys.
|
data/lib/chronos/agent.rb
CHANGED
|
@@ -37,13 +37,16 @@ module Chronos
|
|
|
37
37
|
pipeline_options
|
|
38
38
|
)
|
|
39
39
|
initialize_capture(options)
|
|
40
|
+
@dependency_reporter = options[:dependency_reporter] || Application::DependencyReporter.new(config)
|
|
40
41
|
end
|
|
41
42
|
|
|
42
43
|
def notify(exception, context = {})
|
|
44
|
+
report_dependencies
|
|
43
45
|
@capture.call(exception, context_for_capture(context))
|
|
44
46
|
end
|
|
45
47
|
|
|
46
48
|
def notify_sync(exception, context = {})
|
|
49
|
+
report_dependencies
|
|
47
50
|
@capture.call_sync(exception, context_for_capture(context))
|
|
48
51
|
end
|
|
49
52
|
|
|
@@ -64,9 +67,48 @@ module Chronos
|
|
|
64
67
|
end
|
|
65
68
|
|
|
66
69
|
def record_event(event_type, payload = {}, context = {})
|
|
70
|
+
report_dependencies unless event_type.to_s == "dependencies"
|
|
67
71
|
@telemetry.call(event_type, payload, telemetry_context(context))
|
|
68
72
|
end
|
|
69
73
|
|
|
74
|
+
def report_dependencies
|
|
75
|
+
payload = @dependency_reporter.call
|
|
76
|
+
return false unless payload
|
|
77
|
+
|
|
78
|
+
@telemetry.call("dependencies", payload, {})
|
|
79
|
+
rescue StandardError => error
|
|
80
|
+
@logger.warn("Chronos dependency reporting failed: #{error.class}")
|
|
81
|
+
false
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def record_event_once(key, event_type, payload = {}, context = {})
|
|
85
|
+
execution = @context_store.get
|
|
86
|
+
captured = execution[:__chronos_captured_events] || {}
|
|
87
|
+
return false if captured[key.to_s]
|
|
88
|
+
|
|
89
|
+
captured[key.to_s] = true
|
|
90
|
+
@context_store.set(execution.merge(:__chronos_captured_events => captured))
|
|
91
|
+
record_event(event_type, payload, context)
|
|
92
|
+
rescue StandardError
|
|
93
|
+
false
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
def apm_integration_options
|
|
97
|
+
{
|
|
98
|
+
:enabled => @config.apm_enabled,
|
|
99
|
+
:slow_query_threshold_ms => @config.apm_slow_query_threshold_ms,
|
|
100
|
+
:root_directory => @config.root_directory
|
|
101
|
+
}
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
def external_http_integration_options
|
|
105
|
+
{:enabled => @config.external_http_enabled, :trace_headers => @config.external_http_trace_headers}
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
def cache_integration_options
|
|
109
|
+
{:project_id => @config.project_id, :key_mode => @config.cache_key_mode}
|
|
110
|
+
end
|
|
111
|
+
|
|
70
112
|
# Returns the correlation subset safe for an integration-owned process boundary.
|
|
71
113
|
def propagation_context
|
|
72
114
|
current = context_hash(@context_store.get)
|
|
@@ -104,6 +146,8 @@ module Chronos
|
|
|
104
146
|
end
|
|
105
147
|
|
|
106
148
|
def flush(timeout = DEFAULT_FLUSH_TIMEOUT)
|
|
149
|
+
report_dependencies
|
|
150
|
+
@telemetry.flush
|
|
107
151
|
@delivery_pipeline.flush(timeout)
|
|
108
152
|
rescue StandardError => error
|
|
109
153
|
@logger.warn("Chronos flush failed: #{error.class}")
|
|
@@ -111,6 +155,8 @@ module Chronos
|
|
|
111
155
|
end
|
|
112
156
|
|
|
113
157
|
def close(timeout = DEFAULT_FLUSH_TIMEOUT)
|
|
158
|
+
report_dependencies
|
|
159
|
+
@telemetry.flush
|
|
114
160
|
@delivery_pipeline.close(timeout)
|
|
115
161
|
rescue StandardError => error
|
|
116
162
|
@logger.warn("Chronos close failed: #{error.class}")
|
|
@@ -119,6 +165,7 @@ module Chronos
|
|
|
119
165
|
|
|
120
166
|
def diagnostics
|
|
121
167
|
details = @delivery_pipeline.diagnostics
|
|
168
|
+
details[:apm] = @telemetry.diagnostics
|
|
122
169
|
details[:queue].merge(details)
|
|
123
170
|
end
|
|
124
171
|
|
|
@@ -139,6 +186,8 @@ module Chronos
|
|
|
139
186
|
merged = deep_merge(context_hash(@context_store.get), context_hash(additional))
|
|
140
187
|
merged.delete(:__chronos_captured_exceptions)
|
|
141
188
|
merged.delete("__chronos_captured_exceptions")
|
|
189
|
+
merged.delete(:__chronos_captured_events)
|
|
190
|
+
merged.delete("__chronos_captured_events")
|
|
142
191
|
buffer = merged.delete(:__chronos_breadcrumbs) || merged.delete("__chronos_breadcrumbs")
|
|
143
192
|
if buffer.respond_to?(:to_a)
|
|
144
193
|
merged[:context] = context_hash(merged[:context]).merge("breadcrumbs" => buffer.to_a)
|
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
module Chronos
|
|
2
|
+
module Application
|
|
3
|
+
# Aggregates bounded request, query, job, and external HTTP observations into metric batches.
|
|
4
|
+
#
|
|
5
|
+
# @responsibility Maintain statistics, histograms, breakdown, and local query signals.
|
|
6
|
+
# @motivation Avoid one network event per observation while retaining diagnostic value.
|
|
7
|
+
# @limits Percentiles and heavy correlation remain server-side; state is process-local.
|
|
8
|
+
# @collaborators CaptureTelemetry and immutable Chronos configuration.
|
|
9
|
+
# @thread_safety A mutex protects groups, request trackers, counters, and drains.
|
|
10
|
+
# @compatibility Ruby 2.2.10 through Ruby 2.6; independent of Rails.
|
|
11
|
+
# @example
|
|
12
|
+
# aggregator.record("request", payload, context)
|
|
13
|
+
# batches = aggregator.flush
|
|
14
|
+
# @errors Invalid observations are ignored and never escape to the application.
|
|
15
|
+
# @performance Group, transaction, query, bucket, and batch counts are strictly bounded.
|
|
16
|
+
class ApmAggregator
|
|
17
|
+
include ApmErrorClassifier
|
|
18
|
+
|
|
19
|
+
METRIC_TYPES = %w(request query job external_http).freeze
|
|
20
|
+
def initialize(config)
|
|
21
|
+
@config = config
|
|
22
|
+
@mutex = Mutex.new
|
|
23
|
+
@groups = {}
|
|
24
|
+
@transactions = {}
|
|
25
|
+
@observations = 0
|
|
26
|
+
@dropped_groups = 0
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
def record(event_type, payload = {}, context = {})
|
|
30
|
+
return [] unless @config.apm_enabled
|
|
31
|
+
|
|
32
|
+
@mutex.synchronize do
|
|
33
|
+
type = event_type.to_s
|
|
34
|
+
data = hash(payload)
|
|
35
|
+
execution = hash(context)
|
|
36
|
+
observe_component(type, data, execution)
|
|
37
|
+
add_metric(type, data, execution) if aggregate_metric?(type, data)
|
|
38
|
+
@observations += 1 if aggregate_metric?(type, data)
|
|
39
|
+
@observations >= @config.apm_flush_count ? drain_locked : []
|
|
40
|
+
end
|
|
41
|
+
rescue StandardError
|
|
42
|
+
[]
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def flush
|
|
46
|
+
@mutex.synchronize { drain_locked }
|
|
47
|
+
rescue StandardError
|
|
48
|
+
[]
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
def diagnostics
|
|
52
|
+
@mutex.synchronize do
|
|
53
|
+
{
|
|
54
|
+
"groups" => @groups.length, "dropped_groups" => @dropped_groups,
|
|
55
|
+
"transactions" => @transactions.length,
|
|
56
|
+
"tracked_queries" => @transactions.values.inject(0) do |total, transaction|
|
|
57
|
+
total + transaction["queries"].length
|
|
58
|
+
end
|
|
59
|
+
}
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
private
|
|
64
|
+
|
|
65
|
+
def aggregate_metric?(type, payload)
|
|
66
|
+
METRIC_TYPES.include?(type) && !(type == "request" && payload["kind"].to_s == "view")
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def add_metric(type, payload, context)
|
|
70
|
+
dimensions = dimensions_for(type, payload)
|
|
71
|
+
key = metric_key(type, dimensions)
|
|
72
|
+
aggregate = @groups[key]
|
|
73
|
+
unless aggregate
|
|
74
|
+
if @groups.length >= @config.apm_max_groups
|
|
75
|
+
@dropped_groups += 1
|
|
76
|
+
return
|
|
77
|
+
end
|
|
78
|
+
aggregate = Core::MetricAggregate.new(type, dimensions, @config.apm_histogram_buckets)
|
|
79
|
+
@groups[key] = aggregate
|
|
80
|
+
end
|
|
81
|
+
duration = non_negative(payload["duration_ms"] || payload[:duration_ms])
|
|
82
|
+
status = ["request", "external_http"].include?(type) ? payload["status"] || payload[:status] : nil
|
|
83
|
+
signals = signals_for(type, payload, context, duration)
|
|
84
|
+
breakdown = breakdown_for(type, payload, context, duration)
|
|
85
|
+
aggregate.observe(
|
|
86
|
+
duration, error?(type, payload), breakdown, signals, status
|
|
87
|
+
)
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def observe_component(type, payload, context)
|
|
91
|
+
return if type == "job"
|
|
92
|
+
|
|
93
|
+
trace_id = trace_id(context)
|
|
94
|
+
return if trace_id.empty?
|
|
95
|
+
|
|
96
|
+
transaction = transaction_for(trace_id)
|
|
97
|
+
return unless transaction
|
|
98
|
+
|
|
99
|
+
category = component_category(type, payload)
|
|
100
|
+
return observe_query(transaction, payload) if type == "query" && category.nil?
|
|
101
|
+
return unless category
|
|
102
|
+
|
|
103
|
+
transaction["breakdown_ms"][category] ||= 0.0
|
|
104
|
+
transaction["breakdown_ms"][category] += non_negative(payload["duration_ms"] || payload[:duration_ms])
|
|
105
|
+
observe_query(transaction, payload) if type == "query"
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
def transaction_for(trace_id)
|
|
109
|
+
existing = @transactions[trace_id]
|
|
110
|
+
return existing if existing
|
|
111
|
+
return nil if @transactions.length >= @config.apm_max_groups
|
|
112
|
+
|
|
113
|
+
@transactions[trace_id] = {"breakdown_ms" => {}, "signals" => {}, "queries" => {}}
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
def observe_query(transaction, payload)
|
|
117
|
+
fingerprint = (payload["fingerprint"] || payload[:fingerprint]).to_s
|
|
118
|
+
return if fingerprint.empty?
|
|
119
|
+
|
|
120
|
+
queries = transaction["queries"]
|
|
121
|
+
return unless queries.key?(fingerprint) || queries.length < @config.apm_max_queries_per_request
|
|
122
|
+
|
|
123
|
+
queries[fingerprint] ||= 0
|
|
124
|
+
queries[fingerprint] += 1
|
|
125
|
+
count = queries[fingerprint]
|
|
126
|
+
slow = non_negative(payload["duration_ms"]) >= @config.apm_slow_query_threshold_ms
|
|
127
|
+
increment_signal(transaction, "slow_query") if slow
|
|
128
|
+
increment_signal(transaction, "repeated_query") if count > 1
|
|
129
|
+
increment_signal(transaction, "possible_n_plus_one") if count == @config.apm_n_plus_one_threshold
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
def breakdown_for(type, payload, context, duration)
|
|
133
|
+
explicit = hash(payload["breakdown_ms"] || payload[:breakdown_ms]).dup
|
|
134
|
+
if type == "request"
|
|
135
|
+
transaction = @transactions.delete(trace_id(context))
|
|
136
|
+
explicit = merge_numeric(explicit, transaction && transaction["breakdown_ms"])
|
|
137
|
+
accounted = explicit.values.inject(0.0) { |total, value| total + non_negative(value) }
|
|
138
|
+
explicit["application"] = [duration - accounted, 0.0].max
|
|
139
|
+
elsif type == "query"
|
|
140
|
+
explicit["database"] = duration
|
|
141
|
+
elsif type == "job"
|
|
142
|
+
explicit["queue"] = non_negative(payload["queue_latency_ms"] || payload[:queue_latency_ms])
|
|
143
|
+
explicit["application"] = duration
|
|
144
|
+
end
|
|
145
|
+
explicit
|
|
146
|
+
end
|
|
147
|
+
|
|
148
|
+
def signals_for(type, payload, context, duration)
|
|
149
|
+
return request_signals(context) if type == "request"
|
|
150
|
+
return query_signals(payload, duration) if type == "query"
|
|
151
|
+
|
|
152
|
+
{}
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
def request_signals(context)
|
|
156
|
+
transaction = @transactions[trace_id(context)]
|
|
157
|
+
transaction ? transaction["signals"].dup : {}
|
|
158
|
+
end
|
|
159
|
+
|
|
160
|
+
def query_signals(payload, duration)
|
|
161
|
+
signals = {}
|
|
162
|
+
signals["slow_query"] = 1 if duration >= @config.apm_slow_query_threshold_ms
|
|
163
|
+
signals["long_transaction"] = 1 if long_transaction?(payload, duration)
|
|
164
|
+
error_class = (payload["error_class"] || payload[:error_class]).to_s
|
|
165
|
+
signals["connection_error"] = 1 if error_class =~ /Connection|NoDatabase|Adapter/i
|
|
166
|
+
signals["deadlock"] = 1 if error_class =~ /Deadlock/i
|
|
167
|
+
signals
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
def long_transaction?(payload, duration)
|
|
171
|
+
operation = (payload["operation"] || payload[:operation]).to_s
|
|
172
|
+
name = (payload["name"] || payload[:name]).to_s
|
|
173
|
+
transaction = operation == "BEGIN" || operation == "COMMIT" || name =~ /TRANSACTION/i
|
|
174
|
+
transaction && duration >= @config.apm_long_transaction_threshold_ms
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
def dimensions_for(type, payload)
|
|
178
|
+
names = case type
|
|
179
|
+
when "request" then %w(route method)
|
|
180
|
+
when "query" then %w(adapter operation table fingerprint normalized_query name cached role shard source)
|
|
181
|
+
when "job" then %w(kind class queue status)
|
|
182
|
+
when "external_http" then %w(host method)
|
|
183
|
+
else []
|
|
184
|
+
end
|
|
185
|
+
names.each_with_object({}) do |name, result|
|
|
186
|
+
value = payload[name] || payload[name.to_sym]
|
|
187
|
+
next if value.nil? || value.to_s.empty?
|
|
188
|
+
|
|
189
|
+
result[name] = dimension_value(type, name, value)
|
|
190
|
+
end
|
|
191
|
+
end
|
|
192
|
+
|
|
193
|
+
def dimension_value(type, name, value)
|
|
194
|
+
return value == true if name == "cached"
|
|
195
|
+
return value.to_i if type == "request" && name == "status"
|
|
196
|
+
|
|
197
|
+
text = value.to_s
|
|
198
|
+
text = normalize_route(text) if name == "route"
|
|
199
|
+
text.bytesize > 512 ? text.byteslice(0, 512) : text
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
def normalize_route(route)
|
|
203
|
+
route.split("/").map do |segment|
|
|
204
|
+
segment =~ /\A\d+\z/ || segment =~ /\A[0-9a-f]{8}-[0-9a-f-]{27,}\z/i ? ":id" : segment
|
|
205
|
+
end.join("/")
|
|
206
|
+
end
|
|
207
|
+
|
|
208
|
+
def metric_key(type, dimensions)
|
|
209
|
+
type + "|" + dimensions.keys.sort.map { |key| "#{key}=#{dimensions[key]}" }.join("|")
|
|
210
|
+
end
|
|
211
|
+
|
|
212
|
+
def component_category(type, payload)
|
|
213
|
+
return "database" if type == "query"
|
|
214
|
+
return "view" if type == "request" && payload["kind"].to_s == "view"
|
|
215
|
+
return "cache" if type == "cache"
|
|
216
|
+
return "queue" if type == "job"
|
|
217
|
+
return "external_http" if type == "external_http"
|
|
218
|
+
|
|
219
|
+
nil
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
def trace_id(context)
|
|
223
|
+
(context["trace_id"] || context[:trace_id]).to_s
|
|
224
|
+
end
|
|
225
|
+
|
|
226
|
+
def increment_signal(transaction, name)
|
|
227
|
+
transaction["signals"][name] ||= 0
|
|
228
|
+
transaction["signals"][name] += 1
|
|
229
|
+
end
|
|
230
|
+
|
|
231
|
+
def merge_numeric(left, right)
|
|
232
|
+
hash(right).each do |key, value|
|
|
233
|
+
left[key.to_s] ||= 0.0
|
|
234
|
+
left[key.to_s] += non_negative(value)
|
|
235
|
+
end
|
|
236
|
+
left
|
|
237
|
+
end
|
|
238
|
+
|
|
239
|
+
def drain_locked
|
|
240
|
+
metrics = @groups.values.map(&:to_h)
|
|
241
|
+
if metrics.empty?
|
|
242
|
+
@transactions = {}
|
|
243
|
+
return []
|
|
244
|
+
end
|
|
245
|
+
|
|
246
|
+
dropped = @dropped_groups
|
|
247
|
+
@groups = {}
|
|
248
|
+
@transactions = {}
|
|
249
|
+
@observations = 0
|
|
250
|
+
@dropped_groups = 0
|
|
251
|
+
batches = []
|
|
252
|
+
metrics.each_slice(@config.apm_batch_size) do |slice|
|
|
253
|
+
batches << {"metrics" => slice, "dropped_groups" => batches.empty? ? dropped : 0}
|
|
254
|
+
end
|
|
255
|
+
batches
|
|
256
|
+
end
|
|
257
|
+
|
|
258
|
+
def non_negative(value)
|
|
259
|
+
number = value.to_f
|
|
260
|
+
number < 0.0 ? 0.0 : number
|
|
261
|
+
rescue StandardError
|
|
262
|
+
0.0
|
|
263
|
+
end
|
|
264
|
+
|
|
265
|
+
def hash(value)
|
|
266
|
+
value.is_a?(Hash) ? value : {}
|
|
267
|
+
end
|
|
268
|
+
end
|
|
269
|
+
end
|
|
270
|
+
end
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
module Chronos
|
|
2
|
+
module Application
|
|
3
|
+
# Classifies bounded APM observations without inspecting exception messages.
|
|
4
|
+
module ApmErrorClassifier
|
|
5
|
+
private
|
|
6
|
+
|
|
7
|
+
def error?(type, payload)
|
|
8
|
+
return status_error?(payload) if type == "request"
|
|
9
|
+
return (payload["status"] || payload[:status]).to_s == "failed" if type == "job"
|
|
10
|
+
return external_http_error?(payload) if type == "external_http"
|
|
11
|
+
|
|
12
|
+
!(payload["error_class"] || payload[:error_class]).to_s.empty?
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def external_http_error?(payload)
|
|
16
|
+
return true unless (payload["error_class"] || payload[:error_class]).to_s.empty?
|
|
17
|
+
return true if payload["timeout"] == true || payload[:timeout] == true
|
|
18
|
+
|
|
19
|
+
status_error?(payload)
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def status_error?(payload)
|
|
23
|
+
(payload["status"] || payload[:status]).to_i >= 500
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
end
|
|
@@ -2,10 +2,10 @@ module Chronos
|
|
|
2
2
|
module Application
|
|
3
3
|
# Coordinates bounded non-exception telemetry capture.
|
|
4
4
|
#
|
|
5
|
-
# @responsibility
|
|
6
|
-
# @motivation Keep
|
|
7
|
-
# @limits It handles
|
|
8
|
-
# @collaborators TelemetryEvent, TelemetrySerializer, and DeliveryPipeline.
|
|
5
|
+
# @responsibility Aggregate APM observations or serialize and enqueue integration events.
|
|
6
|
+
# @motivation Keep framework policy and local batching outside transport and domain objects.
|
|
7
|
+
# @limits It handles only the event types declared by TelemetryEvent.
|
|
8
|
+
# @collaborators ApmAggregator, TelemetryEvent, TelemetrySerializer, and DeliveryPipeline.
|
|
9
9
|
# @thread_safety Calls own event state and may execute concurrently.
|
|
10
10
|
# @compatibility Ruby 2.2.10 through Ruby 2.6; independent of Rails.
|
|
11
11
|
# @example
|
|
@@ -20,18 +20,58 @@ module Chronos
|
|
|
20
20
|
@serializer = options[:serializer] || Core::TelemetrySerializer.new(
|
|
21
21
|
config, nil, :max_payload_size => proc { @delivery_pipeline.max_payload_size }
|
|
22
22
|
)
|
|
23
|
+
@aggregator = options[:aggregator] || ApmAggregator.new(config)
|
|
23
24
|
end
|
|
24
25
|
|
|
25
26
|
def call(event_type, payload = {}, context = {})
|
|
26
27
|
return false unless @config.enabled_for_environment?
|
|
27
28
|
return false unless @delivery_pipeline.capture_allowed?(event_type)
|
|
28
29
|
|
|
30
|
+
aggregate = aggregation_enabled?
|
|
31
|
+
batches = aggregate ? @aggregator.record(event_type, payload, context) : []
|
|
32
|
+
return enqueue_batches(batches) if aggregate && aggregate_type?(event_type)
|
|
33
|
+
|
|
29
34
|
event = Core::TelemetryEvent.new(event_type, payload, context)
|
|
30
|
-
@delivery_pipeline.enqueue(@serializer.call(event))
|
|
35
|
+
delivered = @delivery_pipeline.enqueue(@serializer.call(event))
|
|
36
|
+
enqueue_batches(batches)
|
|
37
|
+
delivered
|
|
31
38
|
rescue StandardError => error
|
|
32
39
|
@logger.warn("Chronos telemetry capture failed: #{error.class}")
|
|
33
40
|
false
|
|
34
41
|
end
|
|
42
|
+
|
|
43
|
+
def flush
|
|
44
|
+
enqueue_batches(@aggregator.flush)
|
|
45
|
+
rescue StandardError => error
|
|
46
|
+
@logger.warn("Chronos APM flush failed: #{error.class}")
|
|
47
|
+
false
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def diagnostics
|
|
51
|
+
@aggregator.diagnostics
|
|
52
|
+
rescue StandardError
|
|
53
|
+
{}
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
private
|
|
57
|
+
|
|
58
|
+
def aggregate_type?(event_type)
|
|
59
|
+
ApmAggregator::METRIC_TYPES.include?(event_type.to_s)
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def aggregation_enabled?
|
|
63
|
+
@config.apm_enabled && @delivery_pipeline.event_enabled?("metric_batch")
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def enqueue_batches(batches)
|
|
67
|
+
results = Array(batches).map do |batch|
|
|
68
|
+
next false unless @delivery_pipeline.event_enabled?("metric_batch")
|
|
69
|
+
|
|
70
|
+
event = Core::TelemetryEvent.new("metric_batch", batch)
|
|
71
|
+
@delivery_pipeline.enqueue(@serializer.call(event))
|
|
72
|
+
end
|
|
73
|
+
results.empty? || results.all?
|
|
74
|
+
end
|
|
35
75
|
end
|
|
36
76
|
end
|
|
37
77
|
end
|
|
@@ -67,6 +67,13 @@ module Chronos
|
|
|
67
67
|
false
|
|
68
68
|
end
|
|
69
69
|
|
|
70
|
+
def event_enabled?(event_type)
|
|
71
|
+
@remote_configuration.delivery_enabled?(event_type)
|
|
72
|
+
rescue StandardError => error
|
|
73
|
+
diagnose(error)
|
|
74
|
+
false
|
|
75
|
+
end
|
|
76
|
+
|
|
70
77
|
def max_payload_size
|
|
71
78
|
@remote_configuration.max_payload_size
|
|
72
79
|
end
|