railscope 0.1.2 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6a891dab66be8ea0c685b4edbf8ac6089426d0f8bc48159cefcd8e709afb5f0d
-  data.tar.gz: f0a40a9df1579a6fd37e8c77861d4a07a9f0155794c4440fe23bb7bf55251c05
+  metadata.gz: 407d8ab344bde0dfb68984086462f3aef07114a6d372ce5f6a2a4fb4ca4c5fd1
+  data.tar.gz: 1dff09145783ff05482bf3d9d118401d325d2373e0bdc8fa3a1ae74885713623
 SHA512:
-  metadata.gz: badbe24c5d96dd3ae5be0c51e57db45e0c83cef4a7fdeb79c2640a50c7fd591b5f0ecc973e79fe9ca996361cace6765f2fd0027200733f3750559df8f3b2c3c1
-  data.tar.gz: 72c8ce099532400ebec6cf4b5c22b862ec8ab1fd742b087a69db1d1ead106d4f0a61a0ef2e7030c135359de5d7936d3eff92ef404077e15c92df1bd4566a955d
+  metadata.gz: 3283c7964a1b56b8b89ee354204394d8d5aff9b1d0d7c36d49e2ed45d1d3a0c0382da567e869b9b3b836fbf4a4d9c261beed24efee8931f96e529e8e04d329e0
+  data.tar.gz: 8ef7eaf49c58775642fa9c0cb8a05c451d7f9e07b87315dbdb5c5990eed590c2d66a8d889b40feef213c97cf0ebb6f18edd3606fcc9215dde30e5ae578a43905

data/README.md

@@ -13,7 +13,8 @@ Railscope provides insight into the requests, exceptions, database queries, jobs
 - **Context Correlation** - Link all events from the same request via `request_id`
 - **Sensitive Data Filtering** - Automatic masking of passwords, tokens, and secrets
 - **Dark Mode UI** - Beautiful GitHub-inspired dark interface
-- **Zero Dependencies** - Works with any Rails 7+ application
+- **Storage Backends** - Direct database writes or Redis buffer with batch flush
+- **Zero Dependencies** - Works with any Rails 7+ application (Redis optional)
 
 ## Installation
 
@@ -55,6 +56,9 @@ Create `config/initializers/railscope.rb`:
 
 ```ruby
 Railscope.configure do |config|
+  # Storage backend: :database (default) or :redis (buffer)
+  config.storage_backend = :database
+
   # Data retention (default: 7 days)
   config.retention_days = 30
 
@@ -66,11 +70,49 @@ Railscope.configure do |config|
 end
 ```
 
+### Storage Backends
+
+Railscope supports two storage modes:
+
+| Mode | Write | Read | Requires |
+|------|-------|------|----------|
+| `:database` | Direct INSERT (sync) | PostgreSQL | PostgreSQL |
+| `:redis` | Redis buffer (async) | PostgreSQL | PostgreSQL + Redis |
+
+**`:database`** (default) -- Entries are written directly to PostgreSQL during the request. Simplest setup, no Redis needed.
+
+**`:redis`** -- Entries are buffered in Redis (~0.1ms per write) and batch-flushed to PostgreSQL periodically. Reduces request latency in high-throughput applications.
+
+When using `:redis`, you need to flush the buffer periodically:
+
+```ruby
+# From a background job (Sidekiq, GoodJob, SolidQueue, etc.)
+class RailscopeFlushJob < ApplicationJob
+  queue_as :low
+
+  def perform
+    Railscope::FlushService.call
+  end
+end
+
+# From a cron/scheduler
+every 5.seconds do
+  runner "Railscope::FlushService.call"
+end
+
+# Or via rake
+# $ rake railscope:flush
+```
+
+> **Note:** Entries only appear in the dashboard after being flushed to PostgreSQL.
+
 ### Environment Variables
 
 | Variable | Description | Default |
 |----------|-------------|---------|
 | `RAILSCOPE_ENABLED` | Enable/disable recording | `false` |
+| `RAILSCOPE_STORAGE` | Storage backend (`database` or `redis`) | `database` |
+| `RAILSCOPE_REDIS_URL` | Redis connection URL | Falls back to `REDIS_URL` |
 | `RAILSCOPE_RETENTION_DAYS` | Days to keep entries | `7` |
 
 ## Authorization
@@ -144,6 +186,16 @@ Entries are automatically tagged:
 - **Exceptions**: `exception`, exception class name
 - **Jobs**: `job`, `enqueue`/`perform`, queue name, `failed`
 
+### Rake Tasks
+
+```bash
+# Flush buffered entries from Redis to database (redis mode only)
+rake railscope:flush
+
+# Purge expired entries (older than retention_days)
+rake railscope:purge
+```
+
 ### Purging Old Entries
 
 Run the purge job to remove entries older than `retention_days`:
@@ -212,12 +264,20 @@ Railscope::Entry.expired
 
 Railscope is designed to have minimal impact:
 
-- Events are recorded synchronously but quickly
 - Ignored paths skip all processing
 - Sensitive data filtering is done once before save
 - Purge job removes old entries to control database size
 
+**With `:database` backend:**
+- Entries are written synchronously via INSERT during the request
+
+**With `:redis` backend:**
+- Writes go to Redis (~0.1ms per entry), near-zero impact on request latency
+- `Entry.insert_all` batches up to 1000 records per flush for efficient persistence
+- Flush is safe to run concurrently (Redis `LPOP` is atomic)
+
 For high-traffic production environments, consider:
+- Using `:redis` backend for lower request latency
 - Shorter retention periods
 - Adding high-traffic paths to ignore list
 - Running purge job more frequently

data/app/models/railscope/application_record.rb

@@ -4,9 +4,30 @@ module Railscope
   class ApplicationRecord < ActiveRecord::Base
     self.abstract_class = true
 
-    # Support for separate database connection when RAILSCOPE_DATABASE_URL is configured
-    # This isolates Railscope's writes from the main application database,
-    # preventing lock contention during high-traffic periods.
-    connects_to database: { writing: :railscope, reading: :railscope } if ENV["RAILSCOPE_DATABASE_URL"].present?
+    # Support for separate database connection.
+    # Activated when a "railscope" database is defined in database.yml
+    # or when RAILSCOPE_DATABASE_URL is set.
+    #
+    # Example database.yml:
+    #   development:
+    #     primary:
+    #       <<: *default
+    #       database: myapp_development
+    #       migrations_paths: db/migrate
+    #     railscope:
+    #       <<: *default
+    #       database: myapp_railscope_development
+    #       migrations_paths: db/railscope_migrate
+    #
+    def self.railscope_separate_database?
+      return true if ENV["RAILSCOPE_DATABASE_URL"].present?
+
+      configs = ActiveRecord::Base.configurations.configs_for(env_name: Rails.env)
+      configs.any? { |c| c.name == "railscope" }
+    rescue StandardError
+      false
+    end
+
+    connects_to database: { writing: :railscope, reading: :railscope } if railscope_separate_database?
   end
 end

data/client/src/components/ui/JsonViewer.tsx

@@ -64,14 +64,27 @@ function JsonValue({ value, indent = 0 }: { value: unknown; indent?: number }) {
     }
     return (
       <div>
-        {entries.map(([key, val], i) => (
-          <div key={key} className="flex">
-            <span className="text-blue-400">"{key}"</span>
-            <span className="text-dark-muted mr-2">:</span>
-            <JsonValue value={val} indent={indent + 1} />
-            {i < entries.length - 1 && <span className="text-dark-muted">,</span>}
-          </div>
-        ))}
+        <span className="text-dark-muted">{'{'}</span>
+        <div className="pl-4">
+          {entries.map(([key, val], i) => {
+            const isComplex = val !== null && typeof val === 'object'
+            return (
+              <div key={key}>
+                <span className="text-blue-400">"{key}"</span>
+                <span className="text-dark-muted mr-1">:</span>
+                {isComplex ? (
+                  <JsonValue value={val} indent={indent + 1} />
+                ) : (
+                  <>
+                    <JsonValue value={val} indent={indent + 1} />
+                  </>
+                )}
+                {i < entries.length - 1 && <span className="text-dark-muted">,</span>}
+              </div>
+            )
+          })}
+        </div>
+        <span className="text-dark-muted">{'}'}</span>
       </div>
     )
   }

data/lib/generators/railscope/templates/initializer.rb

@@ -20,6 +20,21 @@ Railscope.configure do |config|
   #
   # config.enabled = true
 
+  # Storage Backend
+  # ---------------
+  # :database - Direct writes to PostgreSQL (simpler, no Redis needed)
+  # :redis    - Buffer in Redis, batch flush to PostgreSQL (faster requests)
+  #
+  # When using :redis, entries are buffered in Redis during requests and
+  # flushed to PostgreSQL periodically via Railscope::FlushService.
+  # You can trigger the flush with:
+  #   - Railscope::FlushService.call (from a job, cron, etc.)
+  #   - rake railscope:flush
+  #
+  # Can also be set via RAILSCOPE_STORAGE env var.
+  #
+  # config.storage_backend = :database
+
   # Retention Period
   # ----------------
   # Number of days to keep entries before purging.

data/lib/railscope/engine.rb

@@ -29,8 +29,23 @@ module Railscope
 
     initializer "railscope.migrations" do |app|
       unless app.root.to_s.match?(root.to_s)
-        config.paths["db/migrate"].expanded.each do |expanded_path|
-          app.config.paths["db/migrate"] << expanded_path
+        if railscope_separate_database?
+          # Separate database: copy migrations to db/railscope_migrate/
+          # so they only run against the railscope database
+          target_dir = app.root.join("db", "railscope_migrate")
+          config.paths["db/migrate"].expanded.each do |source_dir|
+            Dir[File.join(source_dir, "*.rb")].each do |migration|
+              target = target_dir.join(File.basename(migration))
+              next if target.exist?
+
+              FileUtils.mkdir_p(target_dir)
+              FileUtils.cp(migration, target)
+            end
+          end
+        else
+          config.paths["db/migrate"].expanded.each do |expanded_path|
+            app.config.paths["db/migrate"] << expanded_path
+          end
         end
       end
     end
@@ -78,11 +93,20 @@ module Railscope
     end
 
     rake_tasks do
-      # Load sample tasks for testing
+      load Railscope::Engine.root.join("lib/tasks/railscope.rake")
       load Railscope::Engine.root.join("lib/tasks/railscope_sample.rake")
 
       # Subscribe to rake tasks after they're loaded
       Railscope::Subscribers::CommandSubscriber.subscribe
     end
+
+    private
+
+    def railscope_separate_database?
+      configs = ActiveRecord::Base.configurations.configs_for(env_name: Rails.env)
+      configs.any? { |c| c.name == "railscope" }
+    rescue StandardError
+      false
+    end
   end
 end

data/lib/railscope/flush_service.rb

@@ -0,0 +1,92 @@
+# frozen_string_literal: true
+
+module Railscope
+  class FlushService
+    BUFFER_KEY = "railscope:buffer"
+    UPDATES_KEY = "railscope:buffer:updates"
+    BATCH_SIZE = 1000
+
+    def self.call
+      new.call
+    end
+
+    def call
+      total = flush_entries
+      apply_pending_updates
+      total
+    end
+
+    private
+
+    def flush_entries
+      total = 0
+
+      loop do
+        batch = pop_batch(BUFFER_KEY)
+        break if batch.empty?
+
+        entries = batch.map { |json| JSON.parse(json, symbolize_names: true) }
+        batch_insert_to_database(entries)
+        total += entries.size
+      end
+
+      total
+    end
+
+    def apply_pending_updates
+      loop do
+        batch = pop_batch(UPDATES_KEY)
+        break if batch.empty?
+
+        batch.each do |json|
+          update = JSON.parse(json, symbolize_names: true)
+          apply_update(update)
+        end
+      end
+    end
+
+    def apply_update(update)
+      entry = Entry.where(
+        batch_id: update[:batch_id],
+        entry_type: update[:entry_type].to_s
+      ).order(created_at: :desc).first
+      return unless entry
+
+      entry.payload = entry.payload.merge(
+        update[:payload_updates].transform_keys(&:to_s)
+      )
+      entry.save!
+    rescue StandardError => e
+      Rails.logger.debug("[Railscope] Failed to apply buffered update: #{e.message}")
+    end
+
+    def batch_insert_to_database(entries)
+      now = Time.current
+
+      records = entries.map do |entry|
+        {
+          uuid: entry[:uuid],
+          batch_id: entry[:batch_id],
+          family_hash: entry[:family_hash],
+          entry_type: entry[:entry_type],
+          payload: entry[:payload],
+          tags: entry[:tags] || [],
+          should_display_on_index: entry.fetch(:should_display_on_index, true),
+          occurred_at: entry[:occurred_at] || now,
+          created_at: entry[:created_at] || now,
+          updated_at: entry[:updated_at] || now
+        }
+      end
+
+      Entry.insert_all(records)
+    end
+
+    def pop_batch(key)
+      redis.lpop(key, BATCH_SIZE) || []
+    end
+
+    def redis
+      Railscope.redis
+    end
+  end
+end

data/lib/railscope/middleware.rb

@@ -80,20 +80,9 @@ module Railscope
       end
       session_data = extract_session_from_env(context_data[:env])
 
-      # Parse JSON if applicable
+      # Determine response type and handle accordingly (like Telescope)
       content_type = response_headers["Content-Type"] || response_headers["content-type"] || ""
-      looks_like_json = response_body.to_s.start_with?("{", "[")
-      is_json = content_type.include?("application/json") || looks_like_json
-
-      parsed_body = if is_json
-                      begin
-                        JSON.parse(response_body)
-                      rescue StandardError
-                        response_body
-                      end
-                    else
-                      response_body
-                    end
+      parsed_body = parse_response_body(response_body, content_type, context_data[:env])
 
       payload_updates = {
         "response" => parsed_body.presence,
@@ -101,24 +90,129 @@ module Railscope
         "session" => session_data
       }
 
-      storage = Railscope.storage
+      Railscope.storage.update_by_batch(
+        batch_id: context_data[:batch_id],
+        entry_type: "request",
+        payload_updates: payload_updates
+      )
+    rescue StandardError => e
+      Rails.logger.debug("[Railscope] Failed to update entry with response: #{e.message}")
+    end
 
-      if storage.is_a?(Railscope::Storage::Database)
-        entry = Entry.where(batch_id: context_data[:batch_id], entry_type: "request").order(created_at: :desc).first
-        return unless entry
+    def self.parse_response_body(response_body, content_type, env)
+      body_str = response_body.to_s
 
-        entry.payload = entry.payload.merge(payload_updates)
-        entry.save!
+      # Empty response
+      return "Empty Response" if body_str.blank?
 
-      elsif storage.respond_to?(:update_by_batch)
-        storage.update_by_batch(
-          batch_id: context_data[:batch_id],
-          entry_type: "request",
-          payload_updates: payload_updates
-        )
+      # Redirect
+      if env && (location = env["action_dispatch.redirect_url"])
+        return "Redirected to #{location}"
       end
-    rescue StandardError => e
-      Rails.logger.debug("[Railscope] Failed to update entry with response: #{e.message}")
+
+      # JSON response
+      if content_type.include?("application/json") || body_str.match?(/\A\s*[\[{]/)
+        begin
+          return JSON.parse(body_str)
+        rescue StandardError
+          return body_str.truncate(2000)
+        end
+      end
+
+      # Plain text response
+      if content_type.include?("text/plain")
+        return body_str.truncate(2000)
+      end
+
+      # HTML view response — extract template path and data like Telescope
+      extract_view_response(env) || "HTML Response"
+    end
+
+    def self.extract_view_response(env)
+      return nil unless env
+
+      controller = env["action_controller.instance"]
+      return nil unless controller
+
+      view_path = resolve_view_path(controller)
+      return nil unless view_path
+
+      {
+        "view" => view_path,
+        "data" => extract_controller_data(controller)
+      }
+    rescue StandardError
+      nil
+    end
+
+    def self.resolve_view_path(controller)
+      # Try to get the actual rendered template from the controller
+      if controller.respond_to?(:rendered_format, true) || controller.respond_to?(:controller_path)
+        template_path = "app/views/#{controller.controller_path}/#{controller.action_name}"
+
+        # Try to find the actual file with extension
+        if defined?(Rails.root)
+          candidates = Dir.glob(Rails.root.join("#{template_path}.*"))
+          return candidates.first&.sub("#{Rails.root}/", "") if candidates.any?
+        end
+
+        template_path
+      end
+    rescue StandardError
+      nil
+    end
+
+    def self.extract_controller_data(controller)
+      data = {}
+
+      controller.instance_variables.each do |ivar|
+        name = ivar.to_s.delete_prefix("@")
+
+        # Skip internal Rails/controller variables
+        next if name.start_with?("_")
+        next if IGNORED_INSTANCE_VARS.include?(name)
+
+        value = controller.instance_variable_get(ivar)
+        data[name] = safe_serialize(value)
+      end
+
+      data.presence || {}
+    rescue StandardError
+      {}
+    end
+
+    IGNORED_INSTANCE_VARS = %w[
+      request response marked_for_same_origin_verification
+      performed_redirect action_has_layout lookup_context
+      view_context_class current_renderer view_renderer
+      action_name pressed_key action_status response_body
+    ].freeze
+
+    def self.safe_serialize(value, depth: 0)
+      return "..." if depth > 5
+
+      case value
+      when String, Numeric, TrueClass, FalseClass, NilClass
+        value
+      when Symbol
+        value.to_s
+      when Time, DateTime
+        value.iso8601
+      when Date
+        value.to_s
+      when ActiveRecord::Relation
+        value.limit(50).map { |record| safe_serialize(record, depth: depth + 1) }
+      when ActiveRecord::Base
+        Railscope.filter(value.attributes.transform_values { |v| safe_serialize(v, depth: depth + 1) })
+      when Array
+        value.first(50).map { |v| safe_serialize(v, depth: depth + 1) }
+      when Hash
+        value.transform_values { |v| safe_serialize(v, depth: depth + 1) }
+      else
+        value.to_s
+      end
+    rescue StandardError
+      value.class.name
     end
 
     def self.extract_session_from_env(env)

data/lib/railscope/storage/database.rb

@@ -10,6 +10,15 @@ module Railscope
         EntryData.from_active_record(record)
       end
 
+      def update_by_batch(batch_id:, entry_type:, payload_updates:)
+        entry = Entry.where(batch_id: batch_id, entry_type: entry_type).order(created_at: :desc).first
+        return nil unless entry
+
+        entry.payload = entry.payload.merge(payload_updates)
+        entry.save!
+        EntryData.from_active_record(entry)
+      end
+
       def find(uuid)
         record = Entry.find_by(uuid: uuid)
         return nil unless record

data/lib/railscope/storage/redis_buffer.rb

@@ -0,0 +1,95 @@
+# frozen_string_literal: true
+
+module Railscope
+  module Storage
+    class RedisBuffer < Base
+      BUFFER_KEY = "railscope:buffer"
+      UPDATES_KEY = "railscope:buffer:updates"
+      BUFFER_TTL = 4.hours.to_i
+
+      # WRITE → Redis (fast, ~0.1ms)
+      def write(attributes)
+        entry = build_entry(attributes)
+        redis.rpush(BUFFER_KEY, entry.to_json)
+        redis.expire(BUFFER_KEY, BUFFER_TTL)
+        entry
+      end
+
+      # UPDATE → Redis buffer (for response data from middleware)
+      def update_by_batch(batch_id:, entry_type:, payload_updates:)
+        update = {
+          batch_id: batch_id,
+          entry_type: entry_type,
+          payload_updates: payload_updates
+        }
+        redis.rpush(UPDATES_KEY, update.to_json)
+        redis.expire(UPDATES_KEY, BUFFER_TTL)
+      end
+
+      # READ → Database (source of truth)
+      def find(uuid)
+        database_adapter.find(uuid)
+      end
+
+      def all(filters: {}, page: 1, per_page: 25, displayable_only: true)
+        database_adapter.all(filters: filters, page: page, per_page: per_page, displayable_only: displayable_only)
+      end
+
+      def count(filters: {}, displayable_only: true)
+        database_adapter.count(filters: filters, displayable_only: displayable_only)
+      end
+
+      def for_batch(batch_id)
+        database_adapter.for_batch(batch_id)
+      end
+
+      def for_family(family_hash, page: 1, per_page: 25)
+        database_adapter.for_family(family_hash, page: page, per_page: per_page)
+      end
+
+      def family_count(family_hash)
+        database_adapter.family_count(family_hash)
+      end
+
+      def destroy_all!
+        redis.del(BUFFER_KEY, UPDATES_KEY)
+        database_adapter.destroy_all!
+      end
+
+      def destroy_expired!
+        database_adapter.destroy_expired!
+      end
+
+      # READY → needs both Redis AND database
+      def ready?
+        Railscope.redis_available? && database_adapter.ready?
+      end
+
+      private
+
+      def redis
+        Railscope.redis
+      end
+
+      def database_adapter
+        @database_adapter ||= Database.new
+      end
+
+      def build_entry(attributes)
+        now = Time.current
+        EntryData.new(
+          uuid: attributes[:uuid] || SecureRandom.uuid,
+          batch_id: attributes[:batch_id],
+          family_hash: attributes[:family_hash],
+          entry_type: attributes[:entry_type],
+          payload: attributes[:payload] || {},
+          tags: attributes[:tags] || [],
+          should_display_on_index: attributes.fetch(:should_display_on_index, true),
+          occurred_at: attributes[:occurred_at] || now,
+          created_at: attributes[:created_at] || now,
+          updated_at: attributes[:updated_at] || now
+        )
+      end
+    end
+  end
+end

data/lib/railscope/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Railscope
-  VERSION = "0.1.2"
+  VERSION = "0.1.4"
 end

data/lib/railscope.rb

@@ -6,7 +6,8 @@ require_relative "railscope/filter"
 require_relative "railscope/entry_data"
 require_relative "railscope/storage/base"
 require_relative "railscope/storage/database"
-require_relative "railscope/storage/redis_storage"
+require_relative "railscope/storage/redis_buffer"
+require_relative "railscope/flush_service"
 require_relative "railscope/middleware"
 require_relative "railscope/engine"
 
@@ -57,7 +58,7 @@ module Railscope
     def storage
       @storage ||= case storage_backend
                    when STORAGE_REDIS
-                     Storage::RedisStorage.new
+                     Storage::RedisBuffer.new
                    else
                      Storage::Database.new
                    end

data/lib/tasks/railscope.rake

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+namespace :railscope do
+  desc "Flush buffered entries from Redis to database"
+  task flush: :environment do
+    count = Railscope::FlushService.call
+    puts "Flushed #{count} entries"
+  end
+
+  desc "Purge expired entries"
+  task purge: :environment do
+    count = Railscope.storage.destroy_expired!
+    puts "Purged #{count} entries"
+  end
+end