solid_apm 0.8.2 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6218de46aeec80382ad161cffb631016973a2c5f6f4762095b1e250e3424f010
-  data.tar.gz: 7ee929e42611dc2861b1954056a3457aab1de4daff968558140b7cdd113268ad
+  metadata.gz: e3d5be7324da26b5f4720bc0ba9e61717be304286b9ecad84e09adf503095699
+  data.tar.gz: 8969d758c906bb7de44b4024e1f7353bc63f1f4dda0f2ee5631df1e2fa436e2e
 SHA512:
-  metadata.gz: c66d5a85d9f39e9c82d916f8c289e41addbfe4d20543ad671e7ce982be49c9954d08e5553c6b68e1a448143fef109d5dc1a16ba2613bafd1921f8c3d3a1ddebe
-  data.tar.gz: eccd3664d5c31c26408b49db52d6e3c5a377a49861b5500acb18a6f0d56b17422037ae78e90339a6867aaeb7de62f35a040eccb61f4b5118066dda23feeb88af
+  metadata.gz: 1ed2237e30cabd350515555a5e6a66006777835d4e07adcc2dc2758f0fe8e474e931e3da204f44309671923ba1eb1fbcbc32203eaf459a6cebffc2eb3030f3fa
+  data.tar.gz: fb86bb4e14bf08207255ca5e0121b6e50a5ba43e3a4d394d777362df614d47d25e0d54508a66660c01c7ecccf3c53cc00b395981bee6a32378a5903ca64dd268

data/README.md

@@ -5,6 +5,7 @@ Rails engine to manage APM data without using a third party service.
 
 <img src="./docs/img.png" width="600px">
 <img src="./docs/img_1.png" width="600px">
+<img src="./docs/img_2.png" width="600px">
 
 ## Installation
 
@@ -51,6 +52,158 @@ class ApplicationController
 end
 ```
 
+## Configuration
+
+SolidAPM can be configured using the following options in your `config/initializers/solid_apm.rb` file:
+
+### Database Connection
+
+Configure the database connection for SolidAPM:
+
+```ruby
+SolidApm.connects_to = { database: { writing: :solid_apm } }
+```
+
+### ActiveRecord Logger Silencing
+
+Control whether ActiveRecord logger is silenced during SolidAPM operations (default: `true`):
+
+```ruby
+# Disable ActiveRecord logger silencing to see SQL queries in logs
+SolidApm.silence_active_record_logger = false
+```
+
+### Transaction Sampling
+
+Control the sampling rate for transactions using a "1 out of N" approach (default: `1`):
+
+```ruby
+# Sample every transaction (default behavior)
+SolidApm.transaction_sampling = 1
+
+# Sample 1 out of every 2 transactions (50% sampling)
+SolidApm.transaction_sampling = 2
+
+# Sample 1 out of every 5 transactions (20% sampling)  
+SolidApm.transaction_sampling = 5
+
+# Sample 1 out of every 10 transactions (10% sampling)
+SolidApm.transaction_sampling = 10
+```
+
+The sampling is done per-thread using a round-robin counter, ensuring even distribution across requests.
+This is useful for high-traffic applications where you want to reduce the volume of
+APM data while still maintaining representative performance insights.
+
+### Transaction Name Filtering
+
+Filter specific transactions by name using exact string matches or regular expressions:
+
+```ruby
+# Filter specific transactions by exact name
+SolidApm.transaction_filters += ['HomeController#index', /^Rails::HealthController/]
+```
+
+## Data Cleanup
+
+SolidAPM provides a rake task to clean up old transaction data to manage database size over time.
+
+### Manual Cleanup
+
+Clean up transactions older than 1 month (default):
+
+```shell
+bin/rails solid_apm:cleanup
+```
+
+Clean up transactions with custom time periods:
+
+```shell
+# Delete transactions older than 1 week
+bin/rails solid_apm:cleanup[1.week.ago]
+```
+
+### Automated Cleanup with ActiveJob
+
+For production applications, it's recommended to set up automated cleanup.
+
+Example with SolidQueue. Configure recurring cleanup in your `config/recurring.yml`:
+
+```yaml
+solid_apm_cleanup_weekly:
+  class: SolidApm::CleanupJob  
+  cron: "0 3 * * *"   # Every day at 3 AM
+  args: ["1.week.ago"]
+```
+
+## How it works
+
+SolidAPM stores information in the form of transactions, representing incoming HTTP requests which
+listen to a variety of spans (events) from `ActiveSupport::Instrument`. Each span
+saves backtrace information to easily find the source of issues.
+
+### Request transaction
+
+It is based on [ActionDispatch](https://guides.rubyonrails.org/active_support_instrumentation.html#action-dispatch)
+events to start and end a transaction.
+
+A Rack middleware uses [`rack.after_reply`](https://github.blog/engineering/architecture-optimization/performance-at-github-deferring-stats-with-rack-after_reply/)
+to bulk insert transactions and spans after delivering the response, so tracking your application
+doesn't add delay to the client.
+
+### Spans saved
+
+* Request
+* Rendering
+* SQL requests and transactions 
+* Rails cache
+* Net/HTTP
+
+## MCP Server
+
+SolidAPM offers an optional MCP server to allow an AI agent to interact with SolidAPM
+and help identify issues in your application, such as  
+N+1 queries, slow queries and more. The AI agent can analyze and suggest fixes for these issues.
+
+### MCP Server Configuration
+
+The MCP server is only mounted if the [fast-mcp](https://github.com/yjacquin/fast-mcp) gem is installed by your application.
+
+1. Add to your Gemfile:
+
+```ruby
+# Work in progress, plus patch for MCP 2025-06-18 Protocol Revision
+# with StreamableHTTP support
+# https://github.com/yjacquin/fast-mcp/issues/109
+gem 'fast-mcp', branch: 'transport', github: 'Bhacaz/fast-mcp'
+```
+
+2. Configure the MCP server in your `config/initializers/solid_apm.rb`:
+
+```ruby
+SolidApm.mcp_server_config = {
+  name: 'my-app-solid-apm',
+  path: '/solid_apm/mcp',
+  auth_token: Rails.application.credentials.solid_apm[:mcp_auth_token]
+}
+```
+
+3. Test the MCP server by running:
+
+```shell
+curl -X POST http://localhost:3000/solid_apm/mcp \
+  -H "Content-Type: application/json" \
+  -H "Accept: application/json" \
+  -H "Authorization: Bearer <AUTH_TOKEN>" \
+  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}
+```
+
+### MCP usage
+
+1. Add the MCP resource `impactful-transactions` to the context of your prompt.
+2. Prompt example: "Analyze the impactful transactions of my application and suggest improvements, base on the spans details."
+3. Allow the AI agent to use the MCP tool `spans-for-transaction` to retrieve the longest spans for a specific transaction.
+
 ## TODOs
 
 ### Features

data/app/jobs/solid_apm/cleanup_job.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module SolidApm
+  class CleanupJob < ApplicationJob
+    def perform(older_than = '1.month.ago')
+      result = CleanupService.new(older_than: older_than).call
+
+      Rails.logger.info "SolidApm::CleanupJob completed: deleted #{result[:deleted_count]} transactions older than #{result[:cutoff_time]} (#{result[:older_than]})"
+
+      result
+    end
+  end
+end

data/app/models/solid_apm/span_subscriber/action_dispatch.rb

@@ -13,7 +13,7 @@ module SolidApm
           transaction.duration = ((finish.to_f - start.to_f) * 1000).round(6)
           transaction.metadata = {
             params: payload[:request].params.except(:controller, :action),
-            context: SpanSubscriber::Base.context
+            context: SpanSubscriber::Base.context || {}
           }
           SpanSubscriber::Base.context = {}
         end

data/app/models/solid_apm/span_subscriber/base.rb

@@ -1,4 +1,5 @@
 # frozen_string_literal: true
+
 module SolidApm
   module SpanSubscriber
     class Base

data/app/models/solid_apm/transaction.rb

@@ -4,5 +4,14 @@ module SolidApm
     has_many :spans, -> { order(:timestamp, :sequence) }, foreign_key: 'transaction_id', dependent: :delete_all
 
     attribute :uuid, :string, default: -> { SecureRandom.uuid }
+
+    def self.metadata_filter
+      @metadata_filter ||= ActiveSupport::ParameterFilter
+                             .new(Rails.application.config.filter_parameters)
+    end
+
+    def metadata=(value = {})
+      super(self.class.metadata_filter.filter(value))
+    end
   end
 end

data/lib/solid_apm/cleanup_service.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+module SolidApm
+  class CleanupService
+    # Regex to match safe time expressions like "1.week.ago", "2.months.ago", etc.
+    DURATION_PATTERN = /\A(\d+)\.(second|minute|hour|day|week|month|year)s?\.ago\z/.freeze
+    def initialize(older_than: '1.month.ago')
+      @older_than = older_than
+    end
+
+    def call
+      cutoff_time = parse_time_expression(@older_than)
+      deleted_count = Transaction.where(timestamp: ...cutoff_time).destroy_all.size
+
+      {
+        cutoff_time: cutoff_time,
+        deleted_count: deleted_count,
+        older_than: @older_than
+      }
+    end
+
+    private
+
+    def parse_time_expression(expression)
+      match = expression.match(DURATION_PATTERN)
+      raise ArgumentError, 'Invalid time expression format' unless match
+
+      number = match[1].to_i
+      unit = match[2]
+
+      case unit
+      when 'second'
+        number.seconds.ago
+      when 'minute'
+        number.minutes.ago
+      when 'hour'
+        number.hours.ago
+      when 'day'
+        number.days.ago
+      when 'week'
+        number.weeks.ago
+      when 'month'
+        number.months.ago
+      when 'year'
+        number.years.ago
+      else
+        raise ArgumentError, "Unsupported time unit: #{unit}"
+      end
+    end
+  end
+end

data/lib/solid_apm/engine.rb

@@ -10,6 +10,32 @@ module SolidApm
       app.config.assets.precompile += %w( application.css application.js )
     end
 
+    begin
+      # Mount the MCP server only if the main app added the fast_mcp in is Gemfile.
+      require 'fast_mcp'
+      initializer "solid_apm.mount_mcp_server" do |app|
+      mcp_server_config = SolidApm.mcp_server_config.reverse_merge(
+        name: 'solid-apm-mcp',
+        version: '1.0.0',
+        path: '/solid_apm/mcp'
+      )
+
+      FastMcp.mount_in_rails(
+        app,
+        **mcp_server_config
+      ) do |server|
+        app.config.after_initialize do
+          require_relative 'mcp/spans_for_transaction_tool'
+          require_relative 'mcp/impactful_transactions_resource'
+          server.register_resources(SolidApm::Mcp::ImpactfulTransactionsResource)
+          server.register_tools(SolidApm::Mcp::SpansForTransactionTool)
+        end
+      end
+    end
+    rescue LoadError
+      # Ignored
+    end
+
     config.after_initialize do
       SpanSubscriber::Base.subscribe!
     end

data/lib/solid_apm/mcp/impactful_transactions_resource.rb

@@ -0,0 +1,146 @@
+# frozen_string_literal: true
+
+module SolidApm
+  module Mcp
+    class ImpactfulTransactionsResource < FastMcp::Resource
+      uri "solid-apm://impactful-transactions"
+      resource_name "impactful_transactions"
+      mime_type "application/json"
+      description "Returns the most impactful transactions with comprehensive performance metrics"
+
+    def content
+      transactions_with_impact = calculate_impactful_transactions
+
+      result = {
+        metadata: {
+          total_transactions_analyzed: SolidApm::Transaction.where(timestamp: 24.hours.ago..).count,
+          analysis_period: "last 24 hours",
+          ordered_by: "More impactful first: based on P95 latency, transaction frequency, span complexity."
+        },
+        transactions: transactions_with_impact.map do |transaction_data|
+          {
+            id: transaction_data[:transaction].id,
+            uuid: transaction_data[:transaction].uuid,
+            name: transaction_data[:transaction].name,
+            type: transaction_data[:transaction].type,
+            metrics: {
+              p95_latency_ms: transaction_data[:p95_latency],
+              avg_duration_ms: transaction_data[:avg_duration],
+              max_duration_ms: transaction_data[:max_duration],
+              transactions_per_minute: transaction_data[:tpm],
+              max_transactions_per_minute: transaction_data[:max_tpm],
+              avg_spans_per_transaction: transaction_data[:avg_spans],
+              max_spans_per_transaction: transaction_data[:max_spans],
+              total_occurrences: transaction_data[:total_count],
+            },
+            sample_transaction: {
+              uuid: transaction_data[:sample_transaction]&.uuid,
+              duration_ms: transaction_data[:sample_transaction]&.duration,
+              span_count: transaction_data[:sample_span_count],
+              timestamp: transaction_data[:sample_transaction]&.timestamp
+            }
+          }
+        end
+        }
+
+        JSON.generate(result)
+      rescue StandardError => e
+        JSON.generate({ error: e.message, backtrace: e.backtrace.first(5) })
+      end
+
+      private
+
+      def calculate_impactful_transactions
+        # Get transactions from last 24 hours for more relevant data
+        cutoff_time = 24.hours.ago
+
+        # Group transactions by name and type to aggregate metrics
+        transaction_groups = SolidApm::Transaction.includes(:spans).where(timestamp: cutoff_time..).group_by { |t| [t.name, t.type] }
+
+        impact_data = transaction_groups.map do |group_key, transactions| name, type = group_key
+          durations = transactions.map(&:duration).compact
+          span_counts = transactions.map { |t| t.spans.size }
+
+          next if durations.empty?
+
+          # Calculate P95 latency
+          p95_latency = calculate_percentile(durations, 95)
+          avg_duration = durations.sum / durations.size.to_f
+          max_duration = durations.max
+
+          # Calculate transaction frequency metrics
+          total_count = transactions.size
+          time_span_hours = [(Time.current - transactions.map(&:timestamp).min) / 1.hour, 1].max
+          tpm = (total_count / (time_span_hours * 60)).round(2)
+
+          # Calculate max TPM by looking at busiest minute
+          max_tpm = calculate_max_tpm(transactions)
+
+          # Span complexity metrics
+          avg_spans = span_counts.sum / span_counts.size.to_f
+          max_spans = span_counts.max || 0
+
+          # Get a representative sample transaction
+          sample_transaction = transactions.max_by(&:duration)
+          sample_span_count = sample_transaction&.spans&.size || 0
+
+          {
+            transaction: transactions.first, # Representative transaction for metadata
+            p95_latency: p95_latency.round(2),
+            avg_duration: avg_duration.round(2),
+            max_duration: max_duration.round(2),
+            tpm: tpm,
+            max_tpm: max_tpm,
+            avg_spans: avg_spans.round(1),
+            max_spans: max_spans,
+            total_count: total_count,
+            sample_transaction: sample_transaction,
+            sample_span_count: sample_span_count
+          }
+        end.compact
+
+        # Sort by impact score and return top 10
+        impact_data.sort_by do |data|
+          -calculate_impact_score(p95_latency: data[:p95_latency],
+                                  tpm: data[:tpm],
+                                  avg_spans: data[:avg_spans],
+                                  total_count: data[:total_count])
+        end.first(10)
+      end
+
+      def calculate_percentile(array, percentile)
+        return 0 if array.empty?
+
+        sorted = array.sort
+        index = (percentile / 100.0) * (sorted.length - 1)
+
+        if index == index.to_i
+          sorted[index.to_i]
+        else lower = sorted[index.floor]
+        upper = sorted[index.ceil]
+        lower + (upper - lower) * (index - index.floor)
+        end
+      end
+
+      def calculate_max_tpm(transactions)
+        return 0 if transactions.empty?
+
+        # Group by minute and find the busiest minute
+        minute_counts = transactions.group_by { |t| t.timestamp.beginning_of_minute }.values.map(&:size)
+        minute_counts.max || 0
+      end
+
+      def calculate_impact_score(p95_latency:, tpm:, avg_spans:, total_count:)
+        # Weighted impact score calculation
+        # Higher scores indicate more impactful transactions
+
+        latency_score = Math.log([p95_latency, 1].max) * 20 # Log scale for latency
+        frequency_score = Math.log([tpm, 1].max) * 30 # Log scale for frequency
+        complexity_score = Math.log([avg_spans, 1].max) * 15 # Span complexity
+        volume_score = Math.log([total_count, 1].max) * 10 # Total volume
+
+        (latency_score + frequency_score + complexity_score + volume_score).round(2)
+      end
+    end
+  end
+end

data/lib/solid_apm/mcp/spans_for_transaction_tool.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module SolidApm
+  module Mcp
+    class SpansForTransactionTool < FastMcp::Tool
+      tool_name "spans-for-transaction"
+      description "Returns spans for a specific transaction uuid in the APM system, with backtrace information and metadata"
+
+      arguments do
+        required(:transaction_uuid)
+          .filled(:string)
+          .description("The UUID of the transaction to retrieve spans for")
+      end
+
+      def call(transaction_uuid:)
+        transaction = SolidApm::Transaction.find_by!(uuid: transaction_uuid)
+        JSON.generate({
+            transaction: transaction,
+            spans: transaction.spans
+                      }.as_json
+        )
+      rescue StandardError => e
+        JSON.generate({
+          error: e.message,
+          backtrace: e.backtrace.first(5)
+        }.as_json)
+      end
+    end
+  end
+end

data/lib/solid_apm/middleware.rb

@@ -23,22 +23,51 @@ module SolidApm
     def self.call
       transaction = SpanSubscriber::Base.transaction
       SpanSubscriber::Base.transaction = nil
-      if transaction.nil? || transaction.name.start_with?('SolidApm::')
+
+      if transaction.nil? ||
+          transaction_filtered?(transaction.name) ||
+          !Sampler.should_sample?
+
         SpanSubscriber::Base.spans = nil
         return
       end
 
-      ApplicationRecord.transaction do
-        transaction.save!
+      with_silence_logger do
+        ApplicationRecord.transaction do
+          transaction.save!
 
-        SpanSubscriber::Base.spans.each do |span|
-          span[:transaction_id] = transaction.id
+          SpanSubscriber::Base.spans.each do |span|
+            span[:transaction_id] = transaction.id
+          end
+          SolidApm::Span.insert_all SpanSubscriber::Base.spans
         end
-        SolidApm::Span.insert_all SpanSubscriber::Base.spans
       end
       SpanSubscriber::Base.spans = nil
     end
 
+    def self.transaction_filtered?(transaction_name)
+      SolidApm.transaction_filters.any? do |filter|
+        case filter
+        when String
+          transaction_name == filter
+        when Regexp
+          filter.match?(transaction_name)
+        else
+          false
+        end
+      end
+    end
+
+    def self.with_silence_logger
+      if SolidApm.silence_active_record_logger && ActiveRecord::Base.logger
+        ActiveRecord::Base.logger.silence { yield }
+      else
+        yield
+      end
+    end
+
+    # Initialize a new transaction and reset spans
+
     def self.init_transaction
       now = Time.zone.now
       SpanSubscriber::Base.transaction = Transaction.new(

data/lib/solid_apm/sampler.rb

@@ -0,0 +1,12 @@
+module SolidApm
+  class Sampler
+    def self.should_sample?
+      return true if SolidApm.transaction_sampling <= 1
+
+      thread_counter = Thread.current[:solid_apm_counter] ||= 0
+      Thread.current[:solid_apm_counter] = (thread_counter + 1) % SolidApm.transaction_sampling
+
+      Thread.current[:solid_apm_counter] == 0
+    end
+  end
+end

data/lib/solid_apm/version.rb

@@ -1,3 +1,3 @@
 module SolidApm
-  VERSION = "0.8.2"
+  VERSION = "0.10.0"
 end

data/lib/solid_apm.rb

@@ -3,11 +3,23 @@ require 'groupdate'
 require 'active_median'
 require 'apexcharts'
 
-require "solid_apm/version"
-require "solid_apm/engine"
+require 'solid_apm/version'
+require 'solid_apm/engine'
+require 'solid_apm/sampler'
+require 'solid_apm/cleanup_service'
 
 module SolidApm
   mattr_accessor :connects_to
+  mattr_accessor :mcp_server_config, default: {}
+  mattr_accessor :silence_active_record_logger, default: true
+  mattr_accessor :transaction_sampling, default: 1
+  mattr_accessor(
+    :transaction_filters, default: [
+      /^SolidApm::/,
+      /^ActionDispatch::Request::PASS_NOT_FOUND/,
+      'Rails::HealthController#show'
+    ]
+  )
 
   def self.set_context(context)
     SpanSubscriber::Base.context = context

data/lib/tasks/solid_apm_tasks.rake

@@ -1,4 +1,18 @@
-# desc "Explaining what the task does"
-# task :solid_apm do
-#   # Task goes here
-# end
+namespace :solid_apm do
+  desc 'Delete old transactions (default: older than 1 month). Usage: rake solid_apm:cleanup[1.week.ago]'
+  task :cleanup, [:older_than] => :environment do |_task, args|
+    older_than = args[:older_than] || '1.month.ago'
+
+    begin
+      result = SolidApm::CleanupService.new(older_than: older_than).call
+
+      puts "Deleting transactions older than #{result[:cutoff_time]}..."
+      puts "Deleted #{result[:deleted_count]} transactions"
+    rescue StandardError => e
+      puts "Error: #{e.message}"
+      puts "Please provide a valid time expression like '1.week.ago', '2.months.ago', etc."
+      puts 'Supported formats: [number].[unit].ago where unit is: second(s), minute(s), hour(s), day(s), week(s), month(s), year(s)'
+      exit 1
+    end
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: solid_apm
 version: !ruby/object:Gem::Version
-  version: 0.8.2
+  version: 0.10.0
 platform: ruby
 authors:
 - Jean-Francis Bastien
@@ -125,6 +125,7 @@ files:
 - app/controllers/solid_apm/transactions_controller.rb
 - app/helpers/solid_apm/application_helper.rb
 - app/jobs/solid_apm/application_job.rb
+- app/jobs/solid_apm/cleanup_job.rb
 - app/models/solid_apm/application_record.rb
 - app/models/solid_apm/span.rb
 - app/models/solid_apm/span_subscriber/action_dispatch.rb
@@ -145,8 +146,12 @@ files:
 - db/migrate/20240608015633_create_solid_apm_transactions.rb
 - db/migrate/20240608021940_create_solid_apm_spans.rb
 - lib/solid_apm.rb
+- lib/solid_apm/cleanup_service.rb
 - lib/solid_apm/engine.rb
+- lib/solid_apm/mcp/impactful_transactions_resource.rb
+- lib/solid_apm/mcp/spans_for_transaction_tool.rb
 - lib/solid_apm/middleware.rb
+- lib/solid_apm/sampler.rb
 - lib/solid_apm/version.rb
 - lib/tasks/solid_apm_tasks.rake
 homepage: https://github.com/Bhacaz/solid_apm
@@ -170,7 +175,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.7
+rubygems_version: 3.6.9
 specification_version: 4
 summary: SolidApm is a DB base engine for Application Performance Monitoring.
 test_files: []