catpm 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 338d2cf2951113dc9253467fe765f8ec7a2d5c451c67b28dd26e35687d1ca5d7
+  data.tar.gz: 151c5f0739cd6e0e239167347c9bf175ca3583647a18fbe9fbcd5c76cc5b236c
+SHA512:
+  metadata.gz: caa5d925cdc322ab4208ce06c85fe245e969efab1c3bf60c1fb338bfd8e829abd1646c7996cf4836fb9435853598b65b4cf880dee934cc5e43e0cdb7ccd2ccc6
+  data.tar.gz: bedb5e46a7d2a6c6e50cdb567e84e456e7e9d21238086dafd5309e6e45559b241f2a7ee03139afed1f31904e6244914b0fdd1c9a6ac6c592ca4e7bf004746015

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,222 @@
+# Catpm
+
+Lightweight, self-hosted performance monitoring for Rails. Track requests, background jobs, errors, and custom traces — all stored in your existing database. No external services, no Redis, no extra infrastructure.
+
+Catpm is designed for small-to-medium Rails applications where a full APM (Datadog, New Relic) is overkill but flying blind is not an option.
+
+## Features
+
+- **HTTP request tracking** — automatic via Rack middleware, zero configuration
+- **Background job monitoring** — ActiveJob with queue wait time
+- **Custom traces** — instrument any code block with `Catpm.trace` / `Catpm.span`
+- **Segment waterfall** — nested breakdown of SQL, views, cache, HTTP, mailers per request
+- **Error tracking** — fingerprinting, occurrence counting, context circular buffers
+- **Built-in dashboard** — filterable by kind, endpoint drill-down, waterfall visualization
+- **Custom events** — track business events (signups, payments, etc.) with `Catpm.event`
+- **Auto-instrumentation** — service objects (`ApplicationService`, `BaseService`) traced automatically
+- **Multi-database** — PostgreSQL (primary), SQLite (first-class)
+- **Zero dependencies** — only Rails >= 7.1, no Redis or background queues required
+- **Memory-safe** — configurable buffer limits, automatic downsampling with infinite retention
+- **Resilient** — circuit breaker protects your app if the monitoring DB has issues
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem 'catpm'
+```
+
+Run the install generator:
+
+```bash
+bin/rails generate catpm:install
+bin/rails db:migrate
+```
+Visit `/catpm` in your browser — done.
+
+## Usage
+
+### HTTP requests
+
+Tracked automatically. Every controller action is recorded with duration, status, and segment breakdown (SQL queries, view rendering, cache operations, etc.).
+
+### Background jobs
+
+Enable in the initializer:
+
+```ruby
+Catpm.configure do |config|
+  config.instrument_jobs = true
+end
+```
+
+All ActiveJob classes will be tracked with duration and queue wait time.
+
+### Custom traces
+
+Wrap any code block to track it as a standalone operation:
+
+```ruby
+Catpm.trace('Stripe::Charge') do
+  Stripe::Charge.create(amount: 1000, currency: 'usd')
+end
+```
+
+Inside an existing request, `Catpm.span` adds a segment to the waterfall instead of creating a separate trace:
+
+```ruby
+Catpm.span('geocode', type: :external) do
+  Geocoder.search(address)
+end
+```
+
+For cases where a block doesn't work, use the manual API:
+
+```ruby
+span = Catpm.start_trace('long_operation')
+# ... do work ...
+span.finish
+```
+
+### Track non-controller requests
+
+For webhooks, custom Rack endpoints, or anything outside ActionController:
+
+```ruby
+Catpm.track_request(kind: :http, target: 'WebhookController#stripe') do
+  process_webhook(payload)
+end
+```
+
+### Declarative method tracing
+
+Include `SpanHelpers` to trace methods without changing their implementation:
+
+```ruby
+class PaymentService
+  include Catpm::SpanHelpers
+
+  def process(order)
+    # ...
+  end
+  span_method :process
+
+  def self.bulk_charge(users)
+    # ...
+  end
+  span_class_method :bulk_charge
+end
+```
+
+### Auto-instrumentation
+
+Service objects following the `ApplicationService.call` pattern are instrumented automatically — no configuration needed. If your base class has a different name:
+
+```ruby
+Catpm.configure do |config|
+  config.service_base_classes = ['MyServiceBase']
+end
+```
+
+You can also instrument specific methods explicitly:
+
+```ruby
+Catpm.configure do |config|
+  config.auto_instrument_methods = ['Worker#process', 'Gateway.charge']
+end
+```
+
+### Custom events
+
+Track business-level events that aren't tied to performance:
+
+```ruby
+Catpm.event('user.signed_up', plan: 'pro', source: 'landing_page')
+Catpm.event('order.completed', total: 49.99)
+```
+
+Events are aggregated into time buckets with sample payloads preserved. Enable in the initializer:
+
+```ruby
+Catpm.configure do |config|
+  config.events_enabled = true
+end
+```
+
+## Configuration
+
+The generated initializer (`config/initializers/catpm.rb`) documents all options. Key settings:
+
+```ruby
+Catpm.configure do |config|
+  # Only run in production/staging
+  config.enabled = Rails.env.production? || Rails.env.staging?
+
+  # Protect the dashboard
+  config.http_basic_auth_user = ENV['CATPM_USER']
+  config.http_basic_auth_password = ENV['CATPM_PASSWORD']
+  # Or use a custom policy:
+  # config.access_policy = ->(request) { request.env["warden"].user&.admin? }
+
+  # Instrumentation
+  config.instrument_jobs = true           # ActiveJob tracking (default: false)
+  config.instrument_net_http = true       # Outbound HTTP tracking (default: false)
+  config.instrument_middleware_stack = true # Per-middleware segments (default: false)
+
+  # Thresholds
+  config.slow_threshold = 500             # ms — global slow threshold
+  config.slow_threshold_per_kind = {      # Override per kind
+    http: 500,
+    job: 5_000,
+    custom: 1_000
+  }
+
+  # Ignore noisy endpoints
+  config.ignored_targets = [
+    'HealthcheckController#index',
+    '/assets/*',
+  ]
+
+  # Tuning
+  config.max_buffer_memory = 32.megabytes # In-memory buffer limit
+  config.flush_interval = 30              # Seconds between DB flushes
+end
+```
+
+## How it works
+
+Catpm collects events in a thread-safe in-memory buffer. A background thread flushes the buffer to your database every 30 seconds (configurable). Data is aggregated into time buckets with percentile digests (t-digest), so storage grows slowly regardless of traffic volume.
+
+Data is kept forever with progressively coarser resolution:
+- Last hour: 1-minute buckets
+- 1 hour – 24 hours: 5-minute buckets
+- 1 day – 1 week: 1-hour buckets
+- 1 week – 3 months: 1-day buckets
+- Older than 3 months: 1-week buckets
+
+This means storage grows logarithmically — years of history take barely more space than a single week of raw data.
+
+A circuit breaker protects your application — if the monitoring DB fails repeatedly, Catpm stops trying and recovers automatically once the DB is healthy again.
+
+## Database support
+
+Catpm stores all data in its own namespaced tables (`catpm_buckets`, `catpm_samples`, `catpm_errors`, `catpm_event_buckets`, `catpm_event_samples`) using your application's existing database connection.
+
+## Requirements
+
+- Ruby >= 3.1
+- Rails >= 7.1
+
+## Contributing
+
+1. Fork the repo
+2. Create your feature branch (`git checkout -b my-feature`)
+3. Run tests: `bin/rails test`
+4. Run linter: `bin/rubocop`
+5. Commit and push
+6. Open a Pull Request
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,6 @@
+require 'bundler/setup'
+
+APP_RAKEFILE = File.expand_path('test/dummy/Rakefile', __dir__)
+load 'rails/tasks/engine.rake'
+
+require 'bundler/gem_tasks'

data/app/assets/stylesheets/catpm/application.css

@@ -0,0 +1,15 @@
+/*
+ * This is a manifest file that'll be compiled into application.css, which will include all the files
+ * listed below.
+ *
+ * Any CSS and SCSS file within this directory, lib/assets/stylesheets, vendor/assets/stylesheets,
+ * or any plugin's vendor/assets/stylesheets directory can be referenced here using a relative path.
+ *
+ * You're free to add application-wide styles to this file and they'll appear at the bottom of the
+ * compiled file so the styles you add here take precedence over styles defined in any other CSS/SCSS
+ * files in this directory. Styles in this file should be added after the last require_* statement.
+ * It is generally better to create a new file per style scope.
+ *
+ *= require_tree .
+ *= require_self
+ */

data/app/controllers/catpm/application_controller.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module Catpm
+  class ApplicationController < ActionController::Base
+  end
+end

data/app/controllers/catpm/endpoints_controller.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module Catpm
+  class EndpointsController < ApplicationController
+    def show
+      @kind = params[:kind]
+      @target = params[:target]
+      @operation = params[:operation].presence || ''
+
+      # Time range filter
+      @range, period, _bucket_seconds = helpers.parse_range(params[:range], extra_valid: ['all'])
+
+      scope = Catpm::Bucket
+        .where(kind: @kind, target: @target, operation: @operation)
+
+      if @range != 'all'
+        scope = scope.where('bucket_start >= ?', period.ago)
+      end
+
+      @aggregate = scope.pick(
+        'SUM(count)',
+        'SUM(duration_sum)',
+        'MAX(duration_max)',
+        'MIN(duration_min)',
+        'SUM(failure_count)',
+        'SUM(success_count)'
+      )
+
+      @count, @duration_sum, @duration_max, @duration_min, @failure_count, @success_count =
+        @aggregate.map { |v| v || 0 }
+
+      @avg_duration = @count > 0 ? @duration_sum / @count : 0.0
+      @failure_rate = @count > 0 ? @failure_count.to_f / @count : 0.0
+
+      @buckets = scope.order(bucket_start: :desc)
+
+      # Merge all TDigests for combined percentiles
+      @tdigest = @buckets.filter_map(&:tdigest).reduce { |merged, td| merged.merge(td); merged }
+
+      # Aggregate metadata across all buckets
+      @metadata = {}
+      @buckets.each do |b|
+        b.parsed_metadata_sum.each do |k, v|
+          @metadata[k] = (@metadata[k] || 0) + (v.is_a?(Numeric) ? v : 0)
+        end
+      end
+
+      endpoint_samples = Catpm::Sample
+        .joins(:bucket)
+        .where(catpm_buckets: { kind: @kind, target: @target, operation: @operation })
+
+      if @range != 'all'
+        endpoint_samples = endpoint_samples.where('catpm_samples.recorded_at >= ?', period.ago)
+      end
+
+      @slow_samples = endpoint_samples.where(sample_type: 'slow').order(duration: :desc).limit(10)
+      @samples = endpoint_samples.where(sample_type: 'random').order(recorded_at: :desc).limit(10)
+      @error_samples = endpoint_samples.where(sample_type: 'error').order(recorded_at: :desc).limit(10)
+
+      @active_error_count = Catpm::ErrorRecord.unresolved.count
+    end
+  end
+end

data/app/controllers/catpm/errors_controller.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module Catpm
+  class ErrorsController < ApplicationController
+    PER_PAGE = 30
+
+    def index
+      @tab = params[:tab] == 'resolved' ? 'resolved' : 'active'
+      @active_count = Catpm::ErrorRecord.unresolved.count
+      @resolved_count = Catpm::ErrorRecord.resolved.count
+      @active_error_count = @active_count
+
+      scope = if @tab == 'resolved'
+        Catpm::ErrorRecord.resolved
+      else
+        Catpm::ErrorRecord.unresolved
+      end
+
+      @available_kinds = scope.distinct.pluck(:kind).sort
+
+      if params[:kind].present? && @available_kinds.include?(params[:kind])
+        @kind_filter = params[:kind]
+        scope = scope.where(kind: @kind_filter)
+      end
+
+      @sort = %w[error_class occurrences_count last_occurred_at].include?(params[:sort]) ? params[:sort] : 'last_occurred_at'
+      @dir = params[:dir] == 'asc' ? 'asc' : 'desc'
+
+      @total_count = scope.count
+      @page = [params[:page].to_i, 1].max
+      @errors = scope.order(@sort => @dir).offset((@page - 1) * PER_PAGE).limit(PER_PAGE)
+    end
+
+    def show
+      @error = Catpm::ErrorRecord.find(params[:id])
+      @contexts = @error.parsed_contexts
+      @active_error_count = Catpm::ErrorRecord.unresolved.count
+    end
+
+    def resolve
+      error = Catpm::ErrorRecord.find(params[:id])
+      error.resolve!
+      redirect_to catpm.error_path(error), notice: 'Marked as resolved'
+    end
+
+    def unresolve
+      error = Catpm::ErrorRecord.find(params[:id])
+      error.unresolve!
+      redirect_to catpm.error_path(error), notice: 'Reopened'
+    end
+
+    def destroy
+      error = Catpm::ErrorRecord.find(params[:id])
+      error.destroy!
+      redirect_to catpm.errors_path, notice: 'Error deleted'
+    end
+
+    def resolve_all
+      Catpm::ErrorRecord.unresolved.update_all(resolved_at: Time.current)
+      redirect_to catpm.errors_path, notice: 'All errors resolved'
+    end
+  end
+end

data/app/controllers/catpm/events_controller.rb

@@ -0,0 +1,89 @@
+# frozen_string_literal: true
+
+module Catpm
+  class EventsController < ApplicationController
+    PER_PAGE = 25
+
+    def index
+      @range, period, bucket_seconds = helpers.parse_range(params[:range])
+
+      recent_buckets = Catpm::EventBucket.recent(period).to_a
+
+      # Hero metrics
+      @total_events = recent_buckets.sum(&:count)
+      period_minutes = period.to_f / 60
+      @events_per_min = (period_minutes > 0 ? @total_events / period_minutes : 0).round(1)
+
+      # Group by name for table
+      grouped = recent_buckets.group_by(&:name)
+      @unique_names = grouped.keys.size
+
+      events_list = grouped.map do |name, bs|
+        total_count = bs.sum(&:count)
+
+        # Sparkline data for this name
+        slots = {}
+        bs.each do |b|
+          slot_key = (b.bucket_start.to_i / bucket_seconds) * bucket_seconds
+          slots[slot_key] = (slots[slot_key] || 0) + b.count
+        end
+        now_slot = (Time.current.to_i / bucket_seconds) * bucket_seconds
+        sparkline = 60.times.map { |i| slots[now_slot - (59 - i) * bucket_seconds] || 0 }
+
+        {
+          name: name,
+          total_count: total_count,
+          sparkline: sparkline,
+          last_seen: bs.map(&:bucket_start).max
+        }
+      end
+
+      # Sort
+      @sort = %w[name total_count last_seen].include?(params[:sort]) ? params[:sort] : 'total_count'
+      @dir = params[:dir] == 'asc' ? 'asc' : 'desc'
+      events_list = events_list.sort_by { |e| e[@sort.to_sym] || '' }
+      events_list = events_list.reverse if @dir == 'desc'
+
+      @total_event_names = events_list.size
+
+      # Sparkline times for tooltips
+      now_slot = (Time.current.to_i / bucket_seconds) * bucket_seconds
+      @sparkline_times = 60.times.map { |i| Time.at(now_slot - (59 - i) * bucket_seconds).strftime('%H:%M') }
+
+      # Pagination
+      @page = [ params[:page].to_i, 1 ].max
+      @events = events_list.drop((@page - 1) * PER_PAGE).first(PER_PAGE)
+
+      @active_error_count = Catpm::ErrorRecord.unresolved.count
+    end
+
+    def show
+      @name = params[:name]
+      @range, period, bucket_seconds = helpers.parse_range(params[:range])
+
+      recent_buckets = Catpm::EventBucket.by_name(@name).recent(period).to_a
+
+      # Hero metrics
+      @total_count = recent_buckets.sum(&:count)
+      period_minutes = period.to_f / 60
+      @events_per_min = (period_minutes > 0 ? @total_count / period_minutes : 0).round(1)
+      @last_seen = recent_buckets.map(&:bucket_start).max
+
+      # Bar chart data
+      slots = {}
+      recent_buckets.each do |b|
+        slot_key = (b.bucket_start.to_i / bucket_seconds) * bucket_seconds
+        slots[slot_key] = (slots[slot_key] || 0) + b.count
+      end
+
+      now_slot = (Time.current.to_i / bucket_seconds) * bucket_seconds
+      @chart_data = 60.times.map { |i| slots[now_slot - (59 - i) * bucket_seconds] || 0 }
+      @chart_times = 60.times.map { |i| Time.at(now_slot - (59 - i) * bucket_seconds).strftime('%H:%M') }
+
+      # Recent samples
+      @samples = Catpm::EventSample.by_name(@name).order(recorded_at: :desc).limit(20)
+
+      @active_error_count = Catpm::ErrorRecord.unresolved.count
+    end
+  end
+end

data/app/controllers/catpm/samples_controller.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module Catpm
+  class SamplesController < ApplicationController
+    def show
+      @sample = Catpm::Sample.find(params[:id])
+      @bucket = @sample.bucket
+      @context = @sample.parsed_context
+      @segments = @context['segments'] || @context[:segments] || []
+      @summary = @context['segment_summary'] || @context[:segment_summary] || {}
+    end
+  end
+end

data/app/controllers/catpm/status_controller.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+module Catpm
+  class StatusController < ApplicationController
+    PER_PAGE = 25
+
+    def index
+      # Time range (parsed first — everything below uses this)
+      @range, period, bucket_seconds = helpers.parse_range(params[:range])
+
+      recent_buckets = Catpm::Bucket.recent(period).to_a
+
+      # Sparkline data
+      slots = {}
+      recent_buckets.each do |b|
+        slot_key = (b.bucket_start.to_i / bucket_seconds) * bucket_seconds
+        (slots[slot_key] ||= []) << b
+      end
+
+      now_slot = (Time.current.to_i / bucket_seconds) * bucket_seconds
+
+      @sparkline_requests = 60.times.map { |i| bs = slots[now_slot - (59 - i) * bucket_seconds]; bs ? bs.sum(&:count) : 0 }
+      @sparkline_errors = 60.times.map { |i| bs = slots[now_slot - (59 - i) * bucket_seconds]; bs ? bs.sum(&:failure_count) : 0 }
+      @sparkline_durations = 60.times.map do |i|
+        bs = slots[now_slot - (59 - i) * bucket_seconds]
+        next 0.0 unless bs
+        total = bs.sum(&:count)
+        total > 0 ? bs.sum(&:duration_sum) / total : 0.0
+      end
+      @sparkline_times = 60.times.map { |i| Time.at(now_slot - (59 - i) * bucket_seconds).strftime('%H:%M') }
+
+      recent_count = recent_buckets.sum(&:count)
+      recent_failures = recent_buckets.sum(&:failure_count)
+      period_minutes = period.to_f / 60
+      @recent_avg_duration = recent_count > 0 ? (recent_buckets.sum(&:duration_sum) / recent_count).round(1) : 0.0
+      @error_rate = recent_count > 0 ? (recent_failures.to_f / recent_count * 100).round(1) : 0.0
+      @requests_per_min = (recent_count / period_minutes).round(1)
+      @recent_count = recent_count
+
+      # Endpoints — aggregated from the SAME time range as hero metrics
+      grouped = recent_buckets.group_by { |b| [b.kind, b.target, b.operation] }
+
+      endpoints = grouped.map do |key, bs|
+        kind, target, operation = key
+        total_count = bs.sum(&:count)
+        {
+          kind: kind,
+          target: target,
+          operation: operation,
+          total_count: total_count,
+          avg_duration: total_count > 0 ? bs.sum(&:duration_sum) / total_count : 0.0,
+          max_duration: bs.map(&:duration_max).max,
+          total_failures: bs.sum(&:failure_count),
+          last_seen: bs.map(&:bucket_start).max
+        }
+      end
+
+      # Kind filter (URL-based)
+      @available_kinds = endpoints.map { |e| e[:kind] }.uniq.sort
+      @kind_filter = params[:kind] if params[:kind].present? && @available_kinds.include?(params[:kind])
+      endpoints = endpoints.select { |e| e[:kind] == @kind_filter } if @kind_filter
+
+      # Server-side sort
+      @sort = %w[target total_count avg_duration max_duration total_failures last_seen].include?(params[:sort]) ? params[:sort] : 'last_seen'
+      @dir = params[:dir] == 'asc' ? 'asc' : 'desc'
+      endpoints = endpoints.sort_by { |e| e[@sort.to_sym] || '' }
+      endpoints = endpoints.reverse if @dir == 'desc'
+
+      @total_endpoint_count = endpoints.size
+
+      # Pagination
+      @page = [params[:page].to_i, 1].max
+      @endpoints = endpoints.drop((@page - 1) * PER_PAGE).first(PER_PAGE)
+      @endpoint_count = @endpoints.size
+
+      @active_error_count = Catpm::ErrorRecord.unresolved.count
+    end
+  end
+end

data/app/controllers/catpm/system_controller.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module Catpm
+  class SystemController < ApplicationController
+    def index
+      @stats = Catpm.stats
+      @buffer_size = Catpm.buffer&.size || 0
+      @buffer_bytes = Catpm.buffer&.current_bytes || 0
+      @config = Catpm.config
+      @bucket_count = Catpm::Bucket.count
+      @sample_count = Catpm::Sample.count
+      @error_count = Catpm::ErrorRecord.count
+      @oldest_bucket = Catpm::Bucket.minimum(:bucket_start)
+      @active_error_count = Catpm::ErrorRecord.unresolved.count
+    end
+  end
+end