active_record_query_counter 3.2.0 → 3.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: daaa5b12132d82613fb803966073b0b55131f0e47b83c1d05e0e6dca63c3ba98
-  data.tar.gz: ede14e3f86b236a2d39009dfe84d930a0541eea70c1273d7346ff4ac40218154
+  metadata.gz: 499ad971fe53532f23bfa674961133b6c39f850b39d89b102d959be760ba386c
+  data.tar.gz: 0f2eb66f8d2d274ddd2b173fce6f21ba35a7069d398f0f72e03052905f0b9221
 SHA512:
-  metadata.gz: 03c8d4e734928d45f7a95188e4a9dc805ef45dabef52d0144d4323f5100163c2603955d1ffa3e78f22178fec458932ea8f56ee67f346bb7a51eae83264de4cee
-  data.tar.gz: 38e3d4959598bc8a9d8cd80c06b07fa34485e9a812095be7783f7a8161e3bcf0e4f3aaa619a114fe8beac9b4251bfdc78b3e445b67e5e0fe21a505b865bb33b2
+  metadata.gz: 8ead36fbb297cfb37af6ccb6c5b50903c123224c5c0ff72f96cb9ccf4caeac02da8e12b34858f91d7ef4ea153383babe0c6eac37e59bace9491707542ccac48b
+  data.tar.gz: d3042dca0d9c1b0624338367461c0277360a676061dd62dd4f2ba101cfdf7831c67a1382637703e4d91d7da51bfc90e0829b2792ff2b20fecfbcee892b48fff4

data/CHANGELOG.md

@@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 3.3.0
+
+### Added
+
+- `TransactionInfo#query_count` reports the number of queries executed within the transaction.
+- `TransactionInfo#gc_time` and `TransactionInfo#cpu_time` report the GC time and thread CPU time in seconds that elapsed while the transaction was open.
+- `TransactionInfo#idle_time` estimates the time the transaction was open but not spent executing queries, running Ruby code, or paused for garbage collection. A large value indicates the process was waiting on something other than the database (an HTTP request, a sleep, a lock, etc.) while it held the transaction open.
+- The `transaction_time` notification payload now includes `:queries` along with `:gc_time`, `:cpu_time`, and `:idle_time` in milliseconds. The queries are an array of new `ActiveRecordQueryCounter::QueryInfo` objects; each includes the SQL statement, query name, row count, start and end times, and the GC, CPU, and connection setup times measured for the query. Bind parameter values are not included.
+- The full query details for a transaction only live in a transient `ActiveRecordQueryCounter::TransactionDetails` object that populates the `transaction_time` notification payload and is then released. The `TransactionInfo` objects retained for the duration of a `count_queries` block hold only counts and timings, so memory stays flat even when a process creates many implicit transactions (for example, many saves without an explicit transaction).
+
 ## 3.2.0
 
 ### Added

data/README.md

@@ -14,6 +14,9 @@ It measures database usage within a block of code, including:
 - The number of transactions used
 - The total time spent inside transactions
 - The number of transactions that were rolled back
+- The queries executed inside each transaction, with the time each one took
+- The GC time and CPU time spent inside each transaction
+- The idle time inside each transaction spent waiting on things other than the database
 
 This gem is designed to help you:
 
@@ -142,15 +145,22 @@ Triggered when a query exceeds the row_count threshold with the payload:
 Triggered when a transaction exceeds the transaction_time threshold with the payload:
 
 - `:trace` - The stack trace of where the transaction was completed.
+- `:queries` - An array of `ActiveRecordQueryCounter::QueryInfo` objects for the queries executed within the transaction. Each object includes the SQL statement, query name, row count, and timing details. Bind parameter values are not included.
+- `:gc_time` - The GC time that elapsed while the transaction was open (in milliseconds).
+- `:cpu_time` - The thread CPU time spent while the transaction was open (in milliseconds).
+- `:idle_time` - The estimated time the transaction was open but not spent executing queries, running Ruby code, or paused for garbage collection (in milliseconds). A large value indicates the process was waiting on something other than the database (an HTTP request, a sleep, a lock, etc.) while it held the transaction open.
 
 ##### 4. active_record_query_counter.transaction_count notification
 
 Triggered when transactions exceed the transaction_count threshold with the payload:
 
-- `:transactions` - An array of `ActiveRecordQueryCounter::TransactionInfo` objects.
+- `:transactions` - An array of `ActiveRecordQueryCounter::TransactionInfo` objects. Each object includes the number of queries executed within the transaction (`query_count`), the GC time (`gc_time`) and thread CPU time (`cpu_time`) in seconds spent while the transaction was open, and the estimated idle time (`idle_time`) in seconds spent waiting on things other than the database.
 
 The duration of the notification event is the time between when the first transaction was started and the last transaction was completed.
 
+> [!NOTE]
+> The full query details for a transaction are only available in the `transaction_time` notification payload. They are released after the notification fires; the `TransactionInfo` objects retained for the duration of a `count_queries` block hold only counts and timings so that memory stays flat even when many transactions are created. Query details are also only collected while queries are being counted inside a `count_queries` block (or the bundled middleware).
+
 #### Setting Thresholds
 
 Thresholds can be configured **globally** in an initializer:

data/VERSION

@@ -1 +1 @@
-3.2.0
+3.3.0

data/lib/active_record_query_counter/connection_adapter_extension.rb

@@ -3,11 +3,6 @@
 module ActiveRecordQueryCounter
   # Module to prepend to the connection adapter to inject the counting behavior.
   module ConnectionAdapterExtension
-    # Clock used to measure the CPU time consumed by the current thread while a query runs.
-    # It is not available on every platform (e.g. Windows), in which case CPU time is not
-    # measured and is treated as zero.
-    CPU_CLOCK_ID = (Process::CLOCK_THREAD_CPUTIME_ID if defined?(Process::CLOCK_THREAD_CPUTIME_ID))
-
     # Connection adapter methods that establish, verify, or reconnect the underlying database
     # connection. When these run inside a query (for example when a stale connection is
     # re-established after an idle period or a database failover), the wall clock time they
@@ -34,11 +29,12 @@ module ActiveRecordQueryCounter
       # @param sql [String] the SQL statement being executed
       # @param name [String, nil] the name of the query
       # @param binds [Array] the bind parameters
+      # @param connection [Object, nil] the connection adapter the query is being executed on
       # @yield executes the query and returns its result
       # @return [Object] the result of the query
-      def measure_query(sql, name, binds)
+      def measure_query(sql, name, binds, connection = nil)
         gc_start = GC.total_time
-        cpu_start = current_cpu_time
+        cpu_start = ActiveRecordQueryCounter.current_cpu_time
         previous_timer = ActiveRecordQueryCounter.start_connection_timer
         start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
         begin
@@ -48,26 +44,17 @@ module ActiveRecordQueryCounter
         end
         if result.is_a?(ActiveRecord::Result)
           end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-          cpu_time = current_cpu_time - cpu_start
+          cpu_time = ActiveRecordQueryCounter.current_cpu_time - cpu_start
           gc_time = (GC.total_time - gc_start) / 1_000_000_000.0
-          ActiveRecordQueryCounter.add_query(sql, name, binds, result.length, start_time, end_time, gc_time, cpu_time, connection_time)
+          ActiveRecordQueryCounter.add_query(sql, name, binds, result.length, start_time, end_time, gc_time, cpu_time, connection_time, connection: connection)
         end
         result
       end
-
-      private
-
-      # The current thread CPU time in seconds, or 0.0 when the platform does not support it.
-      #
-      # @return [Float]
-      def current_cpu_time
-        CPU_CLOCK_ID ? Process.clock_gettime(CPU_CLOCK_ID) : 0.0
-      end
     end
 
     module InternalExecQuery
       def internal_exec_query(sql, name = nil, binds = [], **kwargs)
-        ConnectionAdapterExtension.measure_query(sql, name, binds) { super }
+        ConnectionAdapterExtension.measure_query(sql, name, binds, self) { super }
       end
     end
 

data/lib/active_record_query_counter/counter.rb

@@ -41,9 +41,14 @@ module ActiveRecordQueryCounter
     # @param trace [Array<String>] the trace of the transaction
     # @param start_time [Float] the monotonic time when the transaction began
     # @param end_time [Float] the monotonic time when the transaction ended
+    # @param query_count [Integer] the number of queries executed within the transaction
+    # @param gc_time [Float] the GC time in seconds that elapsed while the transaction was open
+    # @param cpu_time [Float] the thread CPU time in seconds spent while the transaction was open
+    # @param idle_time [Float] the time in seconds the transaction was open but not spent on
+    #   queries, Ruby code, or garbage collection
     # @return [void]
     # @api private
-    def add_transaction(trace:, start_time:, end_time:)
+    def add_transaction(trace:, start_time:, end_time:, query_count: 0, gc_time: 0.0, cpu_time: 0.0, idle_time: 0.0)
       trace_transactions = @transactions_hash[trace]
       if trace_transactions
         # Memory optimization so that we don't store duplicate traces for every transaction in a loop.
@@ -53,7 +58,7 @@ module ActiveRecordQueryCounter
         @transactions_hash[trace] = trace_transactions
       end
 
-      trace_transactions << TransactionInfo.new(start_time: start_time, end_time: end_time, trace: trace)
+      trace_transactions << TransactionInfo.new(start_time: start_time, end_time: end_time, trace: trace, query_count: query_count, gc_time: gc_time, cpu_time: cpu_time, idle_time: idle_time)
     end
 
     # Return the number of transactions that have been tracked by the counter.

data/lib/active_record_query_counter/query_info.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module ActiveRecordQueryCounter
+  # Data structure for storing information about a query executed within a transaction.
+  # Note that the start and end times are monotonic time and not wall clock time. Bind
+  # parameter values are deliberately not stored since they can contain sensitive data.
+  class QueryInfo
+    attr_reader :sql, :name, :row_count, :start_time, :end_time, :gc_time, :cpu_time, :connection_time
+
+    def initialize(sql:, name:, row_count:, start_time:, end_time:, gc_time: 0.0, cpu_time: 0.0, connection_time: 0.0)
+      @sql = sql
+      @name = name
+      @row_count = row_count
+      @start_time = start_time
+      @end_time = end_time
+      @gc_time = gc_time
+      @cpu_time = cpu_time
+      @connection_time = connection_time
+    end
+
+    # Return the wall clock time spent executing the query.
+    #
+    # @return [Float]
+    def elapsed_time
+      end_time - start_time
+    end
+  end
+end

data/lib/active_record_query_counter/transaction_details.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+module ActiveRecordQueryCounter
+  # Data structure with the full details about a transaction, including the queries executed
+  # within it. Note that the start and end times are monotonic time and not wall clock time.
+  #
+  # This object is transient. It is created when a transaction is recorded to build the
+  # `transaction_time` notification payload and to derive the retained
+  # {ActiveRecordQueryCounter::TransactionInfo}, and is then released so the query details
+  # do not accumulate in memory.
+  class TransactionDetails
+    attr_reader :start_time, :end_time, :trace, :queries, :gc_time, :cpu_time
+
+    def initialize(start_time:, end_time:, trace:, queries: [], gc_time: 0.0, cpu_time: 0.0)
+      @start_time = start_time
+      @end_time = end_time
+      @trace = trace
+      @queries = queries
+      @gc_time = gc_time
+      @cpu_time = cpu_time
+    end
+
+    # Return the time spent in the transaction.
+    #
+    # @return [Float]
+    def elapsed_time
+      end_time - start_time
+    end
+
+    # Return the number of queries executed within the transaction.
+    #
+    # @return [Integer]
+    def query_count
+      queries.size
+    end
+
+    # Estimate the time the transaction was open but not spent executing queries, running Ruby
+    # code, or paused for garbage collection. A large value indicates the process was waiting
+    # on something other than the database (an HTTP request, a sleep, a lock, etc.) while it
+    # held the transaction open.
+    #
+    # The wall clock time of each query is subtracted first since it already includes any GC
+    # and CPU time that occurred while the query ran. Only the GC and CPU time that occurred
+    # outside of queries is then subtracted from the remainder. GC and CPU time normally cover
+    # distinct intervals, but they overlap when this thread triggers a GC; when subtracting
+    # both would drive the result negative, only the larger of the two is subtracted so the
+    # shared interval is removed once (the same heuristic used for the query time). The result
+    # is clamped so it is never negative and never exceeds the elapsed time.
+    #
+    # @return [Float]
+    def idle_time
+      return 0.0 if elapsed_time <= 0.0
+
+      queries_elapsed_time = 0.0
+      queries_gc_time = 0.0
+      queries_cpu_time = 0.0
+      queries.each do |query|
+        queries_elapsed_time += query.elapsed_time
+        queries_gc_time += query.gc_time
+        queries_cpu_time += query.cpu_time
+      end
+
+      remaining_time = elapsed_time - queries_elapsed_time
+      return 0.0 if remaining_time <= 0.0
+
+      other_gc_time = (gc_time - queries_gc_time).clamp(0.0, remaining_time)
+      other_cpu_time = (cpu_time - queries_cpu_time).clamp(0.0, remaining_time)
+
+      idle_time = remaining_time - (other_gc_time + other_cpu_time)
+      idle_time = remaining_time - [other_gc_time, other_cpu_time].max if idle_time.negative?
+      idle_time.clamp(0.0, remaining_time)
+    end
+  end
+end

data/lib/active_record_query_counter/transaction_extension.rb

@@ -16,6 +16,9 @@ module ActiveRecordQueryCounter
     def initialize(...)
       super
       @active_record_query_counter_start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      @active_record_query_counter_gc_start = GC.total_time
+      @active_record_query_counter_cpu_start = ActiveRecordQueryCounter.current_cpu_time
+      @active_record_query_counter_queries = ActiveRecordQueryCounter.register_transaction_queries(connection)
     end
 
     def commit(...)
@@ -44,8 +47,17 @@ module ActiveRecordQueryCounter
       return unless start_time
 
       @active_record_query_counter_start_time = nil
+      ActiveRecordQueryCounter.unregister_transaction_queries(connection)
       end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-      ActiveRecordQueryCounter.add_transaction(start_time, end_time)
+      gc_time = (GC.total_time - @active_record_query_counter_gc_start) / 1_000_000_000.0
+      cpu_time = ActiveRecordQueryCounter.current_cpu_time - @active_record_query_counter_cpu_start
+      ActiveRecordQueryCounter.add_transaction(
+        start_time,
+        end_time,
+        queries: @active_record_query_counter_queries,
+        gc_time: gc_time,
+        cpu_time: cpu_time
+      )
       ActiveRecordQueryCounter.increment_rollbacks if rollback
     end
   end

data/lib/active_record_query_counter/transaction_info.rb

@@ -1,15 +1,24 @@
 # frozen_string_literal: true
 
 module ActiveRecordQueryCounter
-  # Data structure for storing information about a transaction. Note that the start and end
-  # times are monotonic time and not wall clock time.
+  # Data structure for storing the counts and timings of a transaction. Note that the start
+  # and end times are monotonic time and not wall clock time.
+  #
+  # This structure deliberately does not hold the queries executed within the transaction so
+  # that retaining it for the duration of a `count_queries` block stays cheap. The full query
+  # details are only available transiently in the `transaction_time` notification payload
+  # (see {ActiveRecordQueryCounter::TransactionDetails}).
   class TransactionInfo
-    attr_reader :start_time, :end_time, :trace
+    attr_reader :start_time, :end_time, :trace, :query_count, :gc_time, :cpu_time, :idle_time
 
-    def initialize(start_time:, end_time:, trace:)
+    def initialize(start_time:, end_time:, trace:, query_count: 0, gc_time: 0.0, cpu_time: 0.0, idle_time: 0.0)
       @start_time = start_time
       @end_time = end_time
       @trace = trace
+      @query_count = query_count
+      @gc_time = gc_time
+      @cpu_time = cpu_time
+      @idle_time = idle_time
     end
 
     # Return the time spent in the transaction.

data/lib/active_record_query_counter.rb

@@ -4,7 +4,9 @@ require "securerandom"
 
 require_relative "active_record_query_counter/connection_adapter_extension"
 require_relative "active_record_query_counter/counter"
+require_relative "active_record_query_counter/query_info"
 require_relative "active_record_query_counter/thresholds"
+require_relative "active_record_query_counter/transaction_details"
 require_relative "active_record_query_counter/transaction_info"
 require_relative "active_record_query_counter/transaction_extension"
 
@@ -26,6 +28,10 @@ module ActiveRecordQueryCounter
   IGNORED_STATEMENTS = %w[SCHEMA EXPLAIN].freeze
   private_constant :IGNORED_STATEMENTS
 
+  # Clock used to measure the CPU time consumed by the current thread. It is not available on
+  # every platform (e.g. Windows), in which case CPU time is not measured and is treated as zero.
+  CPU_CLOCK_ID = (Process::CLOCK_THREAD_CPUTIME_ID if defined?(Process::CLOCK_THREAD_CPUTIME_ID))
+
   @lock = Mutex.new
   @default_thresholds = Thresholds.new
 
@@ -88,9 +94,11 @@ module ActiveRecordQueryCounter
     # @param cpu_time [Float] the thread CPU time in seconds spent while the query ran
     # @param connection_time [Float] the time in seconds spent establishing, verifying, or
     #   reconnecting the database connection while the query ran
+    # @param connection [Object, nil] the connection adapter the query was executed on; used to
+    #   attach the query to the transaction currently open on that connection, if any
     # @return [void]
     # @api private
-    def add_query(sql, name, binds, row_count, start_time, end_time, gc_time, cpu_time, connection_time = 0.0)
+    def add_query(sql, name, binds, row_count, start_time, end_time, gc_time, cpu_time, connection_time = 0.0, connection: nil)
       return if IGNORED_STATEMENTS.include?(name)
 
       counter = current_counter
@@ -102,6 +110,20 @@ module ActiveRecordQueryCounter
       counter.row_count += row_count
       counter.query_time += query_time
 
+      queries = transaction_queries(connection)
+      if queries
+        queries << QueryInfo.new(
+          sql: sql,
+          name: name,
+          row_count: row_count,
+          start_time: start_time,
+          end_time: end_time,
+          gc_time: gc_time,
+          cpu_time: cpu_time,
+          connection_time: connection_time
+        )
+      end
+
       # The notification duration is the database query time, so the event ends that long after
       # it started rather than at the raw wall clock end time.
       notification_end_time = start_time + query_time
@@ -126,18 +148,50 @@ module ActiveRecordQueryCounter
     #
     # @param start_time [Float] the time the transaction started
     # @param end_time [Float] the time the transaction ended
+    # @param queries [Array<ActiveRecordQueryCounter::QueryInfo>] the queries executed within
+    #   the transaction
+    # @param gc_time [Float] the GC time in seconds that elapsed while the transaction was open
+    # @param cpu_time [Float] the thread CPU time in seconds spent while the transaction was open
     # @return [void]
     # @api private
-    def add_transaction(start_time, end_time)
+    def add_transaction(start_time, end_time, queries: [], gc_time: 0.0, cpu_time: 0.0)
       counter = current_counter
       return unless counter.is_a?(Counter)
 
-      trace = backtrace
-      counter.add_transaction(trace: trace, start_time: start_time, end_time: end_time)
+      # The details object holds the full query information. Only the counts and timings
+      # derived from it are retained on the counter; the details are released when this
+      # method returns so the query data does not accumulate in memory.
+      details = TransactionDetails.new(
+        start_time: start_time,
+        end_time: end_time,
+        trace: backtrace,
+        queries: queries,
+        gc_time: gc_time,
+        cpu_time: cpu_time
+      )
+
+      counter.add_transaction(
+        trace: details.trace,
+        start_time: start_time,
+        end_time: end_time,
+        query_count: details.query_count,
+        gc_time: gc_time,
+        cpu_time: cpu_time,
+        idle_time: details.idle_time
+      )
 
       transaction_time_threshold = counter.thresholds.transaction_time || -1
       if transaction_time_threshold.between?(0, end_time - start_time)
-        send_notification("transaction_time", start_time, end_time, trace: backtrace)
+        send_notification(
+          "transaction_time",
+          start_time,
+          end_time,
+          trace: details.trace,
+          queries: details.queries,
+          gc_time: (gc_time * 1000.0).round(6),
+          cpu_time: (cpu_time * 1000.0).round(6),
+          idle_time: (details.idle_time * 1000.0).round(6)
+        )
       end
     end
 
@@ -152,6 +206,43 @@ module ActiveRecordQueryCounter
       counter.rollback_count += 1
     end
 
+    # The current thread CPU time in seconds, or 0.0 when the platform does not support it.
+    #
+    # @return [Float]
+    # @api private
+    def current_cpu_time
+      CPU_CLOCK_ID ? Process.clock_gettime(CPU_CLOCK_ID) : 0.0
+    end
+
+    # Register a new query list for a transaction that was opened on a connection. Queries
+    # executed on that connection will be appended to the list until it is unregistered.
+    # Registering a connection always replaces any list already registered for it.
+    #
+    # The list is bound to the counter that is active when the transaction is opened. Queries
+    # counted by a different counter (e.g. a nested `count_queries` block) are not appended,
+    # so the recorded query count for the transaction stays consistent with the counter that
+    # recorded the transaction.
+    #
+    # @param connection [Object] the connection adapter the transaction was opened on
+    # @return [Array<ActiveRecordQueryCounter::QueryInfo>] the registered query list
+    # @api private
+    def register_transaction_queries(connection)
+      registry = ActiveSupport::IsolatedExecutionState[:active_record_query_counter_transaction_queries] ||= {}
+      queries = []
+      registry[connection] = {counter: current_counter, queries: queries}
+      queries
+    end
+
+    # Remove the registered query list for a connection when its transaction has ended.
+    #
+    # @param connection [Object] the connection adapter the transaction was opened on
+    # @return [void]
+    # @api private
+    def unregister_transaction_queries(connection)
+      ActiveSupport::IsolatedExecutionState[:active_record_query_counter_transaction_queries]&.delete(connection)
+      nil
+    end
+
     # Begin measuring the time spent establishing, verifying, or reconnecting the database
     # connection for a single query. Returns the timer that was previously in effect so it can
     # be restored by {.stop_connection_timer}; this keeps nested queries (should they ever
@@ -366,6 +457,16 @@ module ActiveRecordQueryCounter
       ActiveSupport::IsolatedExecutionState[:active_record_query_counter_connection_timer] = timer
     end
 
+    # The query list registered for the transaction currently open on a connection, or nil
+    # when the connection is unknown, has no open transaction, or the transaction was opened
+    # under a different counter than the current one.
+    def transaction_queries(connection)
+      return nil if connection.nil?
+
+      entry = ActiveSupport::IsolatedExecutionState[:active_record_query_counter_transaction_queries]&.[](connection)
+      entry[:queries] if entry && entry[:counter].equal?(current_counter)
+    end
+
     def send_notification(name, start_time, end_time, payload = {})
       id = "#{name}-#{SecureRandom.hex}"
       ActiveSupport::Notifications.publish("active_record_query_counter.#{name}", start_time, end_time, id, payload)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: active_record_query_counter
 version: !ruby/object:Gem::Version
-  version: 3.2.0
+  version: 3.3.0
 platform: ruby
 authors:
 - Brian Durand
@@ -37,9 +37,11 @@ files:
 - lib/active_record_query_counter.rb
 - lib/active_record_query_counter/connection_adapter_extension.rb
 - lib/active_record_query_counter/counter.rb
+- lib/active_record_query_counter/query_info.rb
 - lib/active_record_query_counter/rack_middleware.rb
 - lib/active_record_query_counter/sidekiq_middleware.rb
 - lib/active_record_query_counter/thresholds.rb
+- lib/active_record_query_counter/transaction_details.rb
 - lib/active_record_query_counter/transaction_extension.rb
 - lib/active_record_query_counter/transaction_info.rb
 homepage: https://github.com/bdurand/active_record_query_counter