chronos-ruby 1.0.0 → 1.1.1

data/docs/performance.md

@@ -1,6 +1,6 @@
 # Performance
 
-Performance is a functional requirement, but version 1.0 makes no unverified speed claim.
+Performance is a functional requirement, but version 1.1 makes no unverified speed claim.
 
 Current controls:
 
@@ -18,9 +18,11 @@ Current controls:
 - shutdown and flush have caller-controlled timeouts;
 - request context and breadcrumbs have fixed structural and byte limits;
 - Rack middleware never consumes request or response bodies.
-- Rails subscribers copy only small allowlisted field sets and never copy raw SQL or job arguments.
+- Rails subscribers copy only small allowlisted field sets and never deliver raw SQL or job arguments.
 - Sidekiq middleware creates no per-job thread or connection and bounds arguments, collections, nesting, strings, and tags before telemetry capture.
 - APM group, trace, query-fingerprint, histogram, and batch counts are fixed; no APM timer thread is created.
+- active trace trackers survive aggregate drains but expire after `apm_trace_ttl_seconds`; query inspections and transaction connections have independent fixed caps;
+- normalized query analysis is bounded and database inspection is disabled by default; opt-in index/statistics/plan inspection adds database round trips only once per selected fingerprint;
 - outbound HTTP instrumentation uses two clock reads and bounded metadata without body/header traversal;
 - cache normalization is bounded and SHA-256 runs only when explicitly enabled;
 - dependency inventory runs at most once per agent and is capped at 200 loaded specs.
@@ -29,7 +31,21 @@ Current controls:
 
 Run the scripts under `benchmarks/` and record Ruby version, operating system, CPU, warmup, iteration count, median, and dispersion before publishing results. `benchmarks/filtering.rb` measures privacy filtering, `benchmarks/retry_backlog.rb` measures fixed-memory outage behavior, `benchmarks/request_overhead.rb` compares Rack-protocol calls, and `benchmarks/rails_notifications.rb` isolates subscriber normalization overhead.
 
-## Version 1.0.0 release gates
+`benchmarks/query_analysis.rb` compares normalization alone with normalization plus bounded static analysis. It deliberately excludes database inspection because catalog and planner cost must be measured against the actual adapter, schema, statistics, and database host before production enablement.
+
+```bash
+ITERATIONS=100000 WARMUP=5000 bundle exec ruby benchmarks/query_analysis.rb
+```
+
+## Version 1.1.0 release gates
+
+Version 1.1 adds a release-gate benchmark for normalized SQL analysis. Static analysis is calculated once per fingerprint and cached under `apm_query_analysis_max_queries`; database inspection remains disabled in the benchmark because its cost depends on the actual adapter, schema, statistics, network, and database host.
+
+The tag workflow runs query analysis with 1,000 warmup and 10,000 measured iterations, then repeats the existing Rack comparison and 500-event fake-endpoint privacy/load gate. Local preparation evidence is recorded in [Version 1.1 readiness](release-1.1-readiness.md). Results are environment-specific and are not a cross-runtime performance promise.
+
+The local Ruby 2.2.10 preparation run measured 453.981 µs per normalization, 594.862 µs for the first static analysis of a fingerprint, 43.946 µs median incremental Rack work, and 500/500 fake-endpoint deliveries at 378.15 events/s. Static analysis is cached; the 594.862 µs value is not paid for every repeated fingerprint.
+
+## Historical version 1.0.0 release gates
 
 `benchmarks/comparative.rb` compares the same successful Rack fixture without and with Chronos instrumentation. It performs configurable warmup, at least three samples, and reports median plus median absolute deviation. `benchmarks/fake_endpoint_load.rb` sends asynchronous exception events to a local TCP endpoint, verifies the v1 schema marker, ensures the secret key is absent from every payload, and fails on loss, rejection, invalid payload, or timeout.
 

data/docs/privacy-lgpd.md

@@ -15,7 +15,8 @@ Version 0.9 sanitizes exception, framework telemetry, dependency/deploy inventor
 | Unknown Ruby objects | Represented by class name without calling application serialization |
 | Request/response bodies, raw query strings, cookies, authorization headers, raw SQL/binds, cache values, mail bodies, environment variables | Never collected automatically |
 | Sidekiq arguments | Collected automatically, limited before sanitization, then redacted by the common policy |
-| SQL | Comments, quoted/numeric/boolean/null literals removed; binds never read; bounded identifiers remain |
+| SQL | Comments, quoted/numeric/boolean/null literals removed; binds never read; bounded identifiers remain; raw SQL is never transmitted |
+| Query inspection | Disabled by default; local `EXPLAIN` has no `ANALYZE`; only allowlisted plan/catalog metadata is retained |
 | External HTTP | Host/method/status/timing only; URL path/query, Authorization, bodies, headers, and error messages omitted |
 | Cache key | Omitted by default; optional project-scoped SHA-256 hash; cache value never read |
 | Dependencies | Bounded loaded gem names/versions and detected runtime labels; paths and lockfiles omitted |
@@ -39,16 +40,18 @@ Raw cache keys can contain user or business data and are never delivered. Option
 
 ## Rails telemetry
 
-Rails subscribers use per-notification allowlists. SQL events retain only the operation name, cached flag, and duration; cache events omit key and value; mailer events omit addresses and content; Active Job events omit job ID and arguments; view identifiers are reduced to basenames. Controller parameters are sanitized by the normal payload pipeline before queueing.
+Rails subscribers use per-notification allowlists. SQL events retain normalized value-free metadata, bounded analysis, cached flag, duration and optional allowlisted inspection evidence; cache events omit key and value; mailer events omit addresses and content; Active Job events omit arguments; view identifiers are reduced to basenames. Controller parameters are sanitized by the normal payload pipeline before queueing.
 
 ## Sidekiq jobs
 
 The optional Sidekiq middleware is the only version 0.6 integration that automatically reads job arguments. It traverses at most 20 top-level arguments, 20 items per nested collection, four levels, and 512 bytes per string. These structural limits run before the common key and content sanitizer. Trace propagation contains only trace and request identifiers. Do not place credentials or unnecessary personal, health, or financial data in job arguments; configure application-specific blocklist keys and audit representative synthetic payloads before enabling production delivery.
 
-## APM dimensions
+## APM dimensions and query inspection
 
 Metric groups deliberately exclude user IDs, JIDs, raw URLs, request parameters, bind values, exception messages, and cache keys. SQL normalization removes common literal forms and comments before fingerprinting, but retains bounded database identifiers and cannot parse every dialect. Do not encode personal or secret values in schema, table, column, SQL keyword, or operation names. Slow-query source contains a bounded file/line frame under the configured application root, not source-code contents.
 
+Static query analysis is enabled by default and processes only the normalized SQL. Index, statistics, and plan inspection are separate opt-ins. The original SELECT is used only to ask the same local connection for `EXPLAIN` without `ANALYZE`; it is never placed in telemetry. Plan predicates and arbitrary `Extra`/filter text are discarded. Only bounded node type, relation, index, estimated rows, cost and schema-index definitions cross the sanitizer. These identifiers can still disclose business vocabulary, so production enablement requires a representative synthetic payload audit.
+
 ## Rack context and breadcrumbs
 
 The Rack middleware copies only bounded operational fields and parameter hashes that another component already parsed. It does not read `rack.input` or copy `QUERY_STRING`. User context is opt-in through `chronos.user`, and user agent collection is disabled unless the middleware option enables it.

data/docs/protocol-v1.md

@@ -2,6 +2,6 @@
 
 The schemas under `contracts/` are the source of truth for protocol v1. Version `1.0.0` freezes `schema_version: "1.0"` and treats every required field, enum value, privacy exclusion, and maximum as a compatibility contract.
 
-Compatible changes may add optional bounded fields or new event types accepted by the server. Removing or renaming a field, changing its type/meaning, weakening a bound, or making an optional field required needs a new protocol major schema. Authentication remains outside the JSON payload. Contract tests and `script/verify_docs` must pass before a release.
+Compatible changes may add optional bounded fields or new event types accepted by the server. APM v1 metrics can now carry tracking-loss metadata, approximate percentiles, severity counts, at most 20 structured diagnostics, and one bounded query-analysis object. They remain optional in the schema for compatibility even though this agent version emits them. Consumers should persist diagnostic `code` and `severity` as stable model fields and treat English messages/recommendations as display fallback text. Removing or renaming a field, changing its type/meaning, weakening a bound, or adding a required field without coordinated consumer rollout needs a new protocol major schema. Authentication remains outside the JSON payload. Contract tests and `script/verify_docs` must pass before a release.
 
 The explicit integration check uses the normal exception envelope and identifies itself through `context.integration_verification`. Its receiver acknowledgement is governed separately by [`integration-verification-response-v1.schema.json`](../contracts/integration-verification-response-v1.schema.json). A successful HTTP status without a complete, correlated response is not proof of authentication or ingestion.

data/docs/release-1.1-readiness.md

@@ -0,0 +1,51 @@
+# Version 1.1 release evidence
+
+Version `1.1.0` is a backward-compatible minor release of the stable legacy line. It adds bounded SQL diagnostics, optional read-only database inspection, richer query metrics, and safer trace correlation while keeping `schema_version: "1.0"` and the previously validated Ruby/Rails/Sidekiq support matrix.
+
+## Scope decision
+
+The long-term roadmap associates a transitional 1.x line with Ruby 2.7 and Rails 6. This release does not claim that support: no runtime/framework pair becomes `Supported` without dedicated CI, a real example application, fake-endpoint delivery, shutdown, privacy, and integration evidence. Version 1.1.0 is therefore a SemVer-compatible capability release for the current legacy package; the transitional matrix remains future work.
+
+## Candidate evidence
+
+Local release preparation on 2026-08-05 used Ruby 2.2.10 x86_64 on macOS. The tag workflow must reproduce every supported Linux/container gate before RubyGems publication.
+
+| Gate | Local candidate evidence | Tag enforcement |
+|---|---|---|
+| Unit, integration, contract and documentation | 195 examples, zero failures | every legacy core job plus publish `rake` |
+| Ruby style and legacy syntax | 204 Ruby files, zero RuboCop offenses on Ruby 2.2.10 | every legacy core job plus publish `rake` |
+| Query analysis privacy | PostgreSQL/MySQL allowlist specs and final serialized-payload integration test | contract/unit/integration suite |
+| Normalized-query benchmark | 10,000 iterations with warmup; database inspection disabled | `release-readiness` |
+| Legacy Ruby 2.2.10–2.6.10 | locally exercised on 2.2.10; complete result belongs to tag CI | `legacy-core` matrix |
+| Rails 4.2/5.2 applications | no new support claim; complete result belongs to tag CI | `legacy-rails` matrix |
+| Sidekiq 4/5 applications | no new support claim; complete result belongs to tag CI | `legacy-sidekiq` matrix |
+| Repeatable Rack comparison | executed locally with warmup, five samples, median and MAD | `release-readiness` |
+| Fake endpoint load/privacy | 500-event local gate with complete receipt and no secret leakage | `release-readiness` |
+| Package metadata | gem version, contents, checksum and SPDX SBOM verified locally | publish job |
+| Trusted publication | no local push performed | protected RubyGems environment and OIDC |
+
+## Local measurements
+
+The 2026-08-05 Ruby 2.2.10 preparation run produced:
+
+| Measurement | Result |
+|---|---:|
+| SQL normalization, 10,000 iterations after 1,000 warmup | 453.981 µs/iteration |
+| First bounded static analysis, same fixture | 594.862 µs/iteration |
+| Direct Rack median, 10,000 calls × 5 samples | 0.015203 s; MAD 0.000351 s |
+| Chronos Rack median, same fixture | 0.454667 s; MAD 0.014678 s |
+| Median incremental Rack work | 43.946 µs/request |
+| Fake endpoint | 500/500 received, 1.322240 s, 378.15 events/s |
+
+The database-inspection path was not benchmarked because a synthetic adapter would not represent catalog/planner cost. Production opt-in requires an adapter/schema-specific staging measurement.
+
+## Release controls
+
+- the tag must be exactly `v1.1.0` and match `Chronos::VERSION`;
+- `publish` depends on the legacy core, Rails, Sidekiq, and release-readiness jobs;
+- query inspection is disabled by default and never uses `EXPLAIN ANALYZE` or DDL;
+- all new APM v1 properties remain optional in the schema for older consumers;
+- artifacts contain the gem, SHA-256 checksum, and SPDX SBOM;
+- a failed, skipped, or incomplete supported-matrix job blocks publication.
+
+The local artifact is evidence only. Publication must happen through the tag workflow, not with a manual `gem push`.

data/docs/security-review.md

@@ -1,6 +1,6 @@
-# Security review for 1.0.0
+# Security review for 1.1.0
 
-Review date: 2026-07-29. Scope: capture, serialization, transport, integration verification, remote configuration, framework/job integrations, stable release workflow, examples, and fixtures.
+Review date: 2026-08-05. Scope: capture, serialization, transport, query analysis/inspection, integration verification, remote configuration, framework/job integrations, stable release workflow, examples, and fixtures.
 
 Verified by contracts and implementation review:
 
@@ -13,5 +13,9 @@ Verified by contracts and implementation review:
 - Active Job propagation uses a namespaced v1 field containing only bounded trace/request identifiers and does not alter job arguments;
 - fixture privacy is enforced by contract tests and dependency advisories are checked by the security workflow.
 - integration verification accepts only a strict correlated response and never exposes raw receiver bodies, credentials, stack traces, paths, SQL, or internal architecture.
+- normalized query analysis is bounded by fingerprint; raw SQL and binds are never transmitted;
+- database inspection is disabled by default, read-only, limited by duration/fingerprint, and never executes `EXPLAIN ANALYZE` or DDL;
+- plan predicates, arbitrary planner text, database rows, and exception messages are excluded; only allowlisted schema/planner fields and error classes remain;
+- new APM properties are additive/optional under protocol v1, and diagnostics/evidence pass through the common sanitizer and serializer limits.
 
-Residual risks: supported Ruby/Rails versions are end-of-life; in-memory backlog is lost at exit; application filters/ignore rules execute application code; project identifiers and documented job IDs may be personal data in some deployments; package signing is not enabled because no trusted certificate/key lifecycle exists. Stable artifacts use a protected environment, Trusted Publishing, SHA-256 checksums and SPDX SBOMs until signing can be operated safely.
+Residual risks: supported Ruby/Rails versions are end-of-life; opt-in catalog and planner calls add database work and lack one portable timeout across legacy adapters; static index recommendations remain heuristic; in-memory backlog is lost at exit; application filters/ignore rules execute application code; project/schema identifiers and documented job IDs may be personal data in some deployments; package signing is not enabled because no trusted certificate/key lifecycle exists. Stable artifacts use a protected environment, Trusted Publishing, SHA-256 checksums and SPDX SBOMs until signing can be operated safely.

data/docs/troubleshooting.md

@@ -74,6 +74,12 @@ This is intentional. The normalizer removes common literal forms and never reads
 
 Local detectors emit bounded heuristics, not confirmed diagnoses. Repetition can be legitimate and exception class names can be adapter-specific. Confirm the trace in the SaaS and application logs before changing application behavior.
 
+## Index recommendation is inaccurate or inspection reports an error
+
+An `index_candidate` uses normalized SQL only and has low confidence. `missing_index_candidate` means the observed index prefix was absent from the bounded catalog snapshot; it does not evaluate selectivity, write cost, partial/expression indexes, storage, or production data distribution. Validate with a DBA and the actual workload before creating a migration.
+
+Inspection is disabled by default. Enable `apm_query_inspection_enabled` first in staging, keep the fingerprint and duration limits small, then opt into statistics or plans. `query_inspection_failed` contains only the exception class. Unsupported adapters, prepared placeholders, permissions, planner timeouts, or unavailable catalog APIs can cause it. The gem never retries an inspection or falls back to `EXPLAIN ANALYZE`.
+
 ## Context appears missing
 
 The legacy context store is thread-local. A new application-created thread does not inherit context. Establish a new `Chronos.with_context` scope inside that thread, and issue manual notification before the scope exits. For Rack capture, supply user and explicit parameters through the documented environment keys.

data/lib/chronos/agent.rb

@@ -11,7 +11,7 @@ module Chronos
   #   agent.notify(RuntimeError.new("failed"))
   # @errors Capture errors return false; explicit construction requires valid configuration.
   # @performance No worker threads are created until the first asynchronous event.
-  class Agent
+  class Agent # rubocop:disable Metrics/ClassLength
     DEFAULT_FLUSH_TIMEOUT = 5.0
 
     attr_reader :config
@@ -133,7 +133,17 @@ module Chronos
       {
         :enabled => @config.apm_enabled,
         :slow_query_threshold_ms => @config.apm_slow_query_threshold_ms,
-        :root_directory => @config.root_directory
+        :root_directory => @config.root_directory,
+        :query_analysis_enabled => @config.apm_query_analysis_enabled,
+        :query_analysis_max_queries => @config.apm_query_analysis_max_queries,
+        :query_inspection_enabled => @config.apm_query_inspection_enabled,
+        :query_statistics_enabled => @config.apm_query_statistics_enabled,
+        :query_plan_enabled => @config.apm_query_plan_enabled,
+        :query_inspection_min_duration_ms => @config.apm_query_inspection_min_duration_ms,
+        :query_inspection_max_queries => @config.apm_query_inspection_max_queries,
+        :transaction_tracking_enabled => @config.apm_transaction_tracking_enabled,
+        :transaction_max_connections => @config.apm_transaction_max_connections,
+        :transaction_ttl_seconds => @config.apm_trace_ttl_seconds
       }
     end
 

data/lib/chronos/application/apm_aggregator.rb

@@ -13,28 +13,51 @@ module Chronos
     #   batches = aggregator.flush
     # @errors Invalid observations are ignored and never escape to the application.
     # @performance Group, transaction, query, bucket, and batch counts are strictly bounded.
-    class ApmAggregator
+    class ApmAggregator # rubocop:disable Metrics/ClassLength
       include ApmErrorClassifier
 
       METRIC_TYPES = %w(request query job external_http).freeze
-      def initialize(config)
+      QUERY_ERROR_SIGNALS = {
+        "connection_error" => /Connection|NoDatabase|Adapter/i,
+        "deadlock" => /Deadlock/i,
+        "query_timeout" => /StatementTimeout|QueryCanceled|QueryTimeout|Timeout/i,
+        "pool_timeout" => /ConnectionTimeout|PoolTimeout/i,
+        "lock_timeout" => /LockWaitTimeout|LockTimeout/i,
+        "constraint_violation" => /RecordNotUnique|NotNullViolation|ForeignKeyViolation|InvalidForeignKey|Constraint/i
+      }.freeze
+      SIGNAL_DIAGNOSTICS = {
+        "slow_query" => ["warning", "performance", "Query duration reached the slow threshold"].freeze,
+        "long_transaction" => ["warning", "transaction", "Transaction SQL reached the long threshold"].freeze,
+        "connection_error" => ["error", "connection", "Database connection failed"].freeze,
+        "deadlock" => ["error", "locking", "The database reported a deadlock"].freeze,
+        "query_timeout" => ["error", "timeout", "The database query timed out"].freeze,
+        "pool_timeout" => ["error", "connection_pool", "Database connection-pool checkout timed out"].freeze,
+        "lock_timeout" => ["error", "locking", "The database lock wait timed out"].freeze,
+        "constraint_violation" => ["error", "constraint", "The database rejected a constraint"].freeze
+      }.freeze
+      def initialize(config, options = {})
         @config = config
+        @clock = options[:clock] || proc { Time.now.to_f }
         @mutex = Mutex.new
         @groups = {}
         @transactions = {}
         @observations = 0
         @dropped_groups = 0
+        @dropped_trace_trackers = 0
+        @expired_trace_trackers = 0
+        @dropped_query_fingerprints = 0
       end
 
       def record(event_type, payload = {}, context = {})
         return [] unless @config.apm_enabled
 
         @mutex.synchronize do
+          expire_transactions
           type = event_type.to_s
           data = hash(payload)
           execution = hash(context)
-          observe_component(type, data, execution)
-          add_metric(type, data, execution) if aggregate_metric?(type, data)
+          correlation = observe_component(type, data, execution)
+          add_metric(type, data, execution, correlation) if aggregate_metric?(type, data)
           @observations += 1 if aggregate_metric?(type, data)
           @observations >= @config.apm_flush_count ? drain_locked : []
         end
@@ -53,6 +76,9 @@ module Chronos
           {
             "groups" => @groups.length, "dropped_groups" => @dropped_groups,
             "transactions" => @transactions.length,
+            "dropped_trace_trackers" => @dropped_trace_trackers,
+            "expired_trace_trackers" => @expired_trace_trackers,
+            "dropped_query_fingerprints" => @dropped_query_fingerprints,
             "tracked_queries" => @transactions.values.inject(0) do |total, transaction|
               total + transaction["queries"].length
             end
@@ -66,7 +92,7 @@ module Chronos
         METRIC_TYPES.include?(type) && !(type == "request" && payload["kind"].to_s == "view")
       end
 
-      def add_metric(type, payload, context)
+      def add_metric(type, payload, context, correlation)
         dimensions = dimensions_for(type, payload)
         key = metric_key(type, dimensions)
         aggregate = @groups[key]
@@ -80,53 +106,77 @@ module Chronos
         end
         duration = non_negative(payload["duration_ms"] || payload[:duration_ms])
         status = ["request", "external_http"].include?(type) ? payload["status"] || payload[:status] : nil
-        signals = signals_for(type, payload, context, duration)
+        signals = signals_for(type, payload, context, duration, correlation)
+        diagnostics = diagnostics_for(type, payload, context, signals, correlation)
         breakdown = breakdown_for(type, payload, context, duration)
         aggregate.observe(
-          duration, error?(type, payload), breakdown, signals, status
+          duration, error?(type, payload), breakdown, signals,
+          :status => status, :diagnostics => diagnostics,
+          :query_analysis => type == "query" ? hash(payload["analysis"] || payload[:analysis]) : {}
         )
       end
 
       def observe_component(type, payload, context)
-        return if type == "job"
+        return {} if type == "job"
 
         trace_id = trace_id(context)
-        return if trace_id.empty?
+        return {} if trace_id.empty?
 
         transaction = transaction_for(trace_id)
-        return unless transaction
+        return {} unless transaction
+
+        transaction["last_seen_at"] = now
 
         category = component_category(type, payload)
         return observe_query(transaction, payload) if type == "query" && category.nil?
-        return unless category
+        return {} unless category
 
         transaction["breakdown_ms"][category] ||= 0.0
         transaction["breakdown_ms"][category] += non_negative(payload["duration_ms"] || payload[:duration_ms])
-        observe_query(transaction, payload) if type == "query"
+        return observe_query(transaction, payload) if type == "query"
+
+        {}
       end
 
       def transaction_for(trace_id)
         existing = @transactions[trace_id]
         return existing if existing
-        return nil if @transactions.length >= @config.apm_max_groups
+        if @transactions.length >= @config.apm_max_groups
+          @dropped_trace_trackers += 1
+          return nil
+        end
 
-        @transactions[trace_id] = {"breakdown_ms" => {}, "signals" => {}, "queries" => {}}
+        @transactions[trace_id] = {
+          "breakdown_ms" => {}, "signals" => {}, "queries" => {}, "diagnostics" => [],
+          "last_seen_at" => now
+        }
       end
 
       def observe_query(transaction, payload)
         fingerprint = (payload["fingerprint"] || payload[:fingerprint]).to_s
-        return if fingerprint.empty?
+        return {} if fingerprint.empty?
 
         queries = transaction["queries"]
-        return unless queries.key?(fingerprint) || queries.length < @config.apm_max_queries_per_request
+        unless queries.key?(fingerprint) || queries.length < @config.apm_max_queries_per_request
+          @dropped_query_fingerprints += 1
+          return {}
+        end
 
         queries[fingerprint] ||= 0
         queries[fingerprint] += 1
         count = queries[fingerprint]
         slow = non_negative(payload["duration_ms"]) >= @config.apm_slow_query_threshold_ms
         increment_signal(transaction, "slow_query") if slow
-        increment_signal(transaction, "repeated_query") if count > 1
-        increment_signal(transaction, "possible_n_plus_one") if count == @config.apm_n_plus_one_threshold
+        current = {}
+        if count > 1
+          increment_signal(transaction, "repeated_query")
+          current["repeated_query"] = 1
+        end
+        if n_plus_one_candidate?(payload) && count == @config.apm_n_plus_one_threshold
+          increment_signal(transaction, "possible_n_plus_one")
+          current["possible_n_plus_one"] = 1
+        end
+        {"signals" => current, "query_count" => count}
       end
 
       def breakdown_for(type, payload, context, duration)
@@ -145,13 +195,75 @@ module Chronos
         explicit
       end
 
-      def signals_for(type, payload, context, duration)
+      def signals_for(type, payload, context, duration, correlation)
         return request_signals(context) if type == "request"
-        return query_signals(payload, duration) if type == "query"
+        return merge_integer_signals(query_signals(payload, duration), hash(correlation["signals"])) if type == "query"
 
         {}
       end
 
+      def diagnostics_for(type, payload, context, signals, correlation)
+        return request_diagnostics(context) if type == "request"
+        return [] unless type == "query"
+
+        analysis = hash(payload["analysis"] || payload[:analysis])
+        result = Array(analysis["diagnostics"] || analysis[:diagnostics]).first(20)
+        result += signal_diagnostics(signals, correlation)
+        error_class = (payload["error_class"] || payload[:error_class]).to_s
+        unless error_class.empty?
+          result << diagnostic("query_execution_error", "error", "execution",
+                               "The database query raised an exception", "error_class" => error_class[0, 128])
+        end
+        retain_transaction_diagnostics(context, result)
+        result.first(20)
+      end
+
+      def signal_diagnostics(signals, correlation)
+        result = signal_error_diagnostics(signals)
+        if hash(correlation["signals"])["repeated_query"]
+          result << diagnostic("repeated_query", "info", "query_pattern",
+                               "The same query fingerprint repeated in one trace",
+                               "occurrence" => correlation["query_count"])
+        end
+        if hash(correlation["signals"])["possible_n_plus_one"]
+          result << diagnostic("possible_n_plus_one", "warning", "n_plus_one",
+                               "Repeated SELECT reached the N+1 threshold",
+                               "occurrence" => correlation["query_count"])
+          item = diagnostic("n_plus_one_eager_loading", "suggestion", "n_plus_one",
+                            "Review eager loading or batch loading for this query fingerprint")
+          item["recommendation"] = "Use includes, preload, eager_load, or an equivalent bounded batch query " \
+                                   "when semantics allow"
+          result << item
+        end
+        result
+      end
+
+      def signal_error_diagnostics(signals)
+        SIGNAL_DIAGNOSTICS.each_with_object([]) do |(code, values), result|
+          result << diagnostic(code, values[0], values[1], values[2]) if signals[code]
+        end
+      end
+
+      def diagnostic(code, severity, category, message, evidence = {})
+        {"code" => code, "severity" => severity, "category" => category,
+         "message" => message, "evidence" => evidence}
+      end
+
+      def retain_transaction_diagnostics(context, diagnostics)
+        transaction = @transactions[trace_id(context)]
+        return unless transaction
+
+        diagnostics.each do |item|
+          next if transaction["diagnostics"].any? { |existing| existing["code"] == item["code"] }
+          transaction["diagnostics"] << item if transaction["diagnostics"].length < 20
+        end
+      end
+
+      def request_diagnostics(context)
+        transaction = @transactions[trace_id(context)]
+        transaction ? transaction["diagnostics"].dup : []
+      end
+
       def request_signals(context)
         transaction = @transactions[trace_id(context)]
         transaction ? transaction["signals"].dup : {}
@@ -162,16 +274,30 @@ module Chronos
         signals["slow_query"] = 1 if duration >= @config.apm_slow_query_threshold_ms
         signals["long_transaction"] = 1 if long_transaction?(payload, duration)
         error_class = (payload["error_class"] || payload[:error_class]).to_s
-        signals["connection_error"] = 1 if error_class =~ /Connection|NoDatabase|Adapter/i
-        signals["deadlock"] = 1 if error_class =~ /Deadlock/i
+        QUERY_ERROR_SIGNALS.each do |name, pattern|
+          signals[name] = 1 if error_class =~ pattern
+        end
         signals
       end
 
       def long_transaction?(payload, duration)
         operation = (payload["operation"] || payload[:operation]).to_s
         name = (payload["name"] || payload[:name]).to_s
-        transaction = operation == "BEGIN" || operation == "COMMIT" || name =~ /TRANSACTION/i
-        transaction && duration >= @config.apm_long_transaction_threshold_ms
+        transaction = operation == "BEGIN" || operation == "COMMIT" || operation == "ROLLBACK" || name =~ /TRANSACTION/i
+        measured = non_negative(payload["transaction_duration_ms"] || payload[:transaction_duration_ms] || duration)
+        transaction && measured >= @config.apm_long_transaction_threshold_ms
+      end
+
+      def n_plus_one_candidate?(payload)
+        operation = (payload["operation"] || payload[:operation]).to_s
+        cached = payload.key?("cached") ? payload["cached"] : payload[:cached]
+        operation == "SELECT" && cached != true
+      end
+
+      def merge_integer_signals(left, right)
+        result = left.dup
+        right.each { |key, value| result[key.to_s] = result.fetch(key.to_s, 0) + value.to_i }
+        result
       end
 
       def dimensions_for(type, payload)
@@ -238,23 +364,47 @@ module Chronos
 
       def drain_locked
         metrics = @groups.values.map(&:to_h)
-        if metrics.empty?
-          @transactions = {}
-          return []
-        end
+        return [] if metrics.empty?
 
         dropped = @dropped_groups
         @groups = {}
-        @transactions = {}
         @observations = 0
         @dropped_groups = 0
+        tracking = {
+          "active_traces" => @transactions.length,
+          "dropped_trace_trackers" => @dropped_trace_trackers,
+          "expired_trace_trackers" => @expired_trace_trackers,
+          "dropped_query_fingerprints" => @dropped_query_fingerprints
+        }
+        @dropped_trace_trackers = 0
+        @expired_trace_trackers = 0
+        @dropped_query_fingerprints = 0
         batches = []
         metrics.each_slice(@config.apm_batch_size) do |slice|
+          slice.first["tracking"] = batches.empty? ? tracking : empty_tracking
           batches << {"metrics" => slice, "dropped_groups" => batches.empty? ? dropped : 0}
         end
         batches
       end
 
+      def expire_transactions
+        threshold = now - @config.apm_trace_ttl_seconds.to_f
+        expired = @transactions.keys.select { |key| @transactions[key]["last_seen_at"].to_f < threshold }
+        expired.each { |key| @transactions.delete(key) }
+        @expired_trace_trackers += expired.length
+      end
+
+      def empty_tracking
+        {"active_traces" => @transactions.length, "dropped_trace_trackers" => 0,
+         "expired_trace_trackers" => 0, "dropped_query_fingerprints" => 0}
+      end
+
+      def now
+        @clock.call.to_f
+      rescue StandardError
+        Time.now.to_f
+      end
+
       def non_negative(value)
         number = value.to_f
         number < 0.0 ? 0.0 : number

data/lib/chronos/configuration/apm_validation.rb

@@ -9,7 +9,7 @@ module Chronos
     # @thread_safety Reads one mutable configuration instance without shared state.
     # @compatibility Ruby 2.2.10 through Ruby 2.6.
     # @errors Invalid values become messages and never raise from validation.
-    module ApmConfigurationValidation
+    module ApmConfigurationValidation # rubocop:disable Metrics/ModuleLength
       private
 
       def apm_errors
@@ -32,6 +32,24 @@ module Chronos
         unless positive_integer?(apm_max_queries_per_request)
           errors << "apm_max_queries_per_request must be a positive integer"
         end
+        errors.concat(query_capacity_errors)
+        errors
+      end
+
+      def query_capacity_errors
+        errors = []
+        unless apm_query_analysis_max_queries.is_a?(Integer) &&
+               apm_query_analysis_max_queries >= 1 && apm_query_analysis_max_queries <= 500
+          errors << "apm_query_analysis_max_queries must be between 1 and 500"
+        end
+        unless apm_query_inspection_max_queries.is_a?(Integer) &&
+               apm_query_inspection_max_queries >= 1 && apm_query_inspection_max_queries <= 100
+          errors << "apm_query_inspection_max_queries must be between 1 and 100"
+        end
+        unless apm_transaction_max_connections.is_a?(Integer) &&
+               apm_transaction_max_connections >= 1 && apm_transaction_max_connections <= 500
+          errors << "apm_transaction_max_connections must be between 1 and 500"
+        end
         errors
       end
 
@@ -46,9 +64,41 @@ module Chronos
         unless apm_n_plus_one_threshold.is_a?(Integer) && apm_n_plus_one_threshold >= 2
           errors << "apm_n_plus_one_threshold must be an integer greater than or equal to 2"
         end
+        errors.concat(query_threshold_errors)
         errors
       end
 
+      def query_threshold_errors
+        errors = []
+        unless positive_number?(apm_trace_ttl_seconds)
+          errors << "apm_trace_ttl_seconds must be greater than zero"
+        end
+        unless non_negative_number?(apm_query_inspection_min_duration_ms)
+          errors << "apm_query_inspection_min_duration_ms must be zero or greater"
+        end
+        query_analysis_boolean_attributes.each do |name, value|
+          errors << "#{name} must be true or false" unless boolean?(value)
+        end
+        if (apm_query_statistics_enabled || apm_query_plan_enabled) && !apm_query_inspection_enabled
+          errors << "query statistics and plans require apm_query_inspection_enabled"
+        end
+        errors
+      end
+
+      def query_analysis_boolean_attributes
+        {
+          :apm_query_analysis_enabled => apm_query_analysis_enabled,
+          :apm_query_inspection_enabled => apm_query_inspection_enabled,
+          :apm_query_statistics_enabled => apm_query_statistics_enabled,
+          :apm_query_plan_enabled => apm_query_plan_enabled,
+          :apm_transaction_tracking_enabled => apm_transaction_tracking_enabled
+        }
+      end
+
+      def non_negative_number?(value)
+        value.is_a?(Numeric) && value >= 0
+      end
+
       def increasing_positive_numbers?(values)
         return false unless values.is_a?(Array) && !values.empty? && values.length <= 19
         return false unless values.all? { |value| positive_number?(value) }