logtail-rails 0.2.15 → 0.3.1

data/example-project/README.md

@@ -9,23 +9,23 @@ You can download the example project from GitHub directly or you can clone it to
 bundle install
 ```
 
-This will install all dependencies listed in the `Gemfile.lock` file.
+This will install all dependencies listed in the `Gemfile`. The example runs on Rails 7.1 and newer.
 
-Then replace `<SOURCE_TOKEN>` and `<INGESTING_HOST>` in `config/application.rb` with your actual source token and ingesting host which you can find by going to [Sources](https://telemetry.betterstack.com/team/0/sources) -> Configure in Better Stack.
+`config/application.rb` reads your source token and ingesting host from the environment. You can find them by going to [Sources](https://telemetry.betterstack.com/team/0/sources) -> Configure in Better Stack.
 
 ```ruby
 config.logger = Logtail::Logger.create_default_logger(
-  "<YOUR_ACTUAL_SOURCE_TOKEN>",
-  ingesting_host: "in.logs.betterstack.com",
+  ENV["BETTER_STACK_SOURCE_TOKEN"],
+  ingesting_host: ENV["BETTER_STACK_INGESTING_HOST"],
 )
 ```
 
 ## Run the example project
  
-To run the example application, run the following command:
+To run the example application, run the following command with your source token and ingesting host:
 
 ```bash
-rails server
+BETTER_STACK_SOURCE_TOKEN=<source_token> BETTER_STACK_INGESTING_HOST=<ingesting_host> bin/rails server
 ```
 
 This will open a local server [127.0.0.1:3000.](http://127.0.0.1:3000/) On the main page, click the "Let's go!" button to generate test logs.
@@ -37,7 +37,9 @@ All done!
 Log into your Logtail account to check your logs.
 ```
 
-This will create a total of 6 different logs, each corresponding to a different log level and one with additional structured data. You can review these logs in Logtail.
+This will create a log for each log level, one with tags and one with additional structured data, and on Rails 8.1 and newer two events reported with `Rails.event`, next to the request, controller and template logs of the Rails integration. Click "Raise an exception" on the main page to see how an exception raised in a controller action is logged. You can review these logs in Better Stack.
+
+Stop the server with Ctrl-C. The logger sends what it still holds before the server exits.
 
 # Logging
 
@@ -185,12 +187,53 @@ This will generate the following JSON output:
 }
 ```
 
-We will automatically add the information about the current user to each log if you're using Ruby on Rails and the Devise gem.
+We will automatically add the information about the current user to each log if you're using Ruby on Rails with Devise (or any other Warden-based authentication) or with Clearance.
+
+### Context for every log line of a request
+
+The request and response lines (`Started GET "/" for 127.0.0.1` and `Completed 200 OK in 12.3ms`) are logged by a Rack middleware, before the request reaches your controller. Context set from inside a controller is therefore attached to the logs from the controller action on, but not to these two lines. To attach context to every log line of a request, set it in a Rack middleware instead.
+
+The user context from Devise or Clearance is set this way, by the `Logtail::Integrations::Rack::UserContext` middleware. If you authenticate differently, give it a lambda that finds the user in the Rack environment and returns a hash, or `nil` when nobody is signed in:
+
+```ruby
+# config/initializers/logtail.rb
+Logtail::Integrations::Rack::UserContext.custom_user_hash = lambda do |rack_env|
+  user_id = rack_env["rack.session"]["user_id"]
+  user = User.find_by(id: user_id) if user_id
+  user && { id: user.id, email: user.email }
+end
+```
+
+The hash is logged as `context.user` on every log line of the request, including the request and response lines. The lambda runs on every request, so keep it cheap. It can read anything from the Rack environment, for example an API token from `rack_env["HTTP_AUTHORIZATION"]`.
 
-If you're not using Devise or you want to log some additional information for every request your Rails app handles, you can easily implement this using Rails' `around_action` in your application controller. A simple implementation could look like this:
+For any other per-request context (a tenant, an API client, a GraphQL operation name), write your own middleware that wraps the request in `Logtail.with_context` and insert it before the middleware that logs the request and response lines:
 
 ```ruby
-class ApplicationController < ActionController::Base
+# config/initializers/logtail.rb
+class LogtailTenantContext
+  def initialize(app)
+    @app = app
+  end
+
+  def call(env)
+    tenant = env["HTTP_X_TENANT"]
+    return @app.call(env) unless tenant
+
+    Logtail.with_context(tenant: { name: tenant }) { @app.call(env) }
+  end
+end
+
+Rails.application.config.middleware.insert_before Logtail::Integrations::Rack::HTTPEvents, LogtailTenantContext
+```
+
+Inserting it before `Logtail::Integrations::Rack::HTTPEvents` also places it after your authentication middleware (such as Warden), so the session and the signed-in user are already available in `env`.
+
+### Context from a controller
+
+If the user is only known once your controller runs (for example, you authenticate from a token in a `before_action`), you can still set the context with Rails' `around_action`. It covers the logs from the controller action and everything it calls, but not the request and response lines, which are logged before the action runs. A simple implementation could look like this:
+
+```ruby
+class ApplicationController < ActionController::Base # or ActionController::API
   around_action :with_logtail_context
 
   private
@@ -202,7 +245,7 @@ class ApplicationController < ActionController::Base
         yield
       end
     end
-    
+
     def user_context
       Logtail::Contexts::User.new(
         id: current_user.id,

data/example-project/app/controllers/example_controller.rb

@@ -38,13 +38,20 @@ class ExampleController < ApplicationController
         )
 
         # Since Rails 8.1, you can also use the Rails.event to send events with structured data to Better Stack.
-        Rails.event.notify("My first event", user_id: 123, email: "user@example.com")
-        
-        # You can add context to all events
-        Rails.event.set_context(request_id: "abc123", shop_id: 456)
-        # And tags specific events
-        Rails.event.tagged("api") do
-            Rails.event.notify("My tagged event with additional context", user_id: 123, email: "user@example.com")
+        if Rails.respond_to?(:event)
+            Rails.event.notify("My first event", user_id: 123, email: "user@example.com")
+
+            # You can add context to all events
+            Rails.event.set_context(request_id: "abc123", shop_id: 456)
+            # And tags specific events
+            Rails.event.tagged("api") do
+                Rails.event.notify("My tagged event with additional context", user_id: 123, email: "user@example.com")
+            end
         end
     end
+
+    # Exceptions raised in an action are sent to Better Stack too, with their class, message and backtrace
+    def raise_error
+        raise ArgumentError, "Something went wrong, this action raised an exception!"
+    end
 end

data/example-project/app/views/example/index.html.erb

@@ -1,3 +1,5 @@
 <h1>Hello from Logtail</h1>
 <p>Click the button below to send some example logs</p>
-<%= button_to "Let's go!", send_logs_path, method: :get %>
+<%= button_to "Let's go!", send_logs_path, method: :get %>
+<p>Or raise an exception in a controller action</p>
+<%= button_to "Raise an exception", raise_error_path, method: :get %>

data/example-project/config/application.rb

@@ -8,8 +8,8 @@ Bundler.require(*Rails.groups)
 
 module ExampleProject
   class Application < Rails::Application
-    # Initialize configuration defaults for originally generated Rails version.
-    config.load_defaults 8.1
+    # Initialize configuration defaults of Rails 7.1, the oldest Rails this example runs on.
+    config.load_defaults 7.1
 
     # Configuration for the application, engines, and railties goes here.
     #
@@ -19,9 +19,10 @@ module ExampleProject
     # config.time_zone = "Central Time (US & Canada)"
     # config.eager_load_paths << Rails.root.join("extras")
 
+    # Send the logs to your Better Stack source, its token and ingesting host are read from the environment
     config.logger = Logtail::Logger.create_default_logger(
-      "<SOURCE_TOKEN>",
-      ingesting_host: "<INGESTING_HOST>",
+      ENV["BETTER_STACK_SOURCE_TOKEN"],
+      ingesting_host: ENV["BETTER_STACK_INGESTING_HOST"],
     )
   end
 end

data/example-project/config/routes.rb

@@ -1,6 +1,7 @@
 Rails.application.routes.draw do
   # Define your application routes per the DSL in https://guides.rubyonrails.org/routing.html
   get '/send_logs', to: 'example#send_logs', as: 'send_logs'
+  get '/raise_error', to: 'example#raise_error', as: 'raise_error'
 
   # Defines the root path route ("/")
   root 'example#index'  

data/lib/logtail-rails/error_event.rb

@@ -18,9 +18,11 @@ module Logtail
         # We determine this when the app loads to avoid the overhead on a per request basis.
         EXCEPTION_WRAPPER_TAKES_CLEANER = defined?(::ActionDispatch::ExceptionWrapper) &&
           !::ActionDispatch::ExceptionWrapper.instance_methods.include?(:env)
-        # config.action_dispatch.log_rescued_responses (Rails 7.0+), as ActionDispatch::DebugExceptions
-        # reads it. This gem silences its logging and logs the exception here instead.
+        # config.action_dispatch.log_rescued_responses (Rails 7.0+) and
+        # config.action_dispatch.debug_exception_log_level (Rails 7.1+), as ActionDispatch::DebugExceptions
+        # reads them. This gem silences its logging and logs the exception here instead.
         LOG_RESCUED_RESPONSES_KEY = "action_dispatch.log_rescued_responses".freeze
+        DEBUG_EXCEPTION_LOG_LEVEL_KEY = "action_dispatch.debug_exception_log_level".freeze
 
         def call(env)
           begin
@@ -38,7 +40,7 @@ module Logtail
         def log_exception(env, exception)
           return if !log_exception?(env, exception)
 
-          Config.instance.logger.fatal do
+          Config.instance.logger.add(env[DEBUG_EXCEPTION_LOG_LEVEL_KEY] || ::Logger::FATAL) do
             backtrace = extract_backtrace(env, exception)
             Events::Error.new(
               name: exception.class.name,

data/lib/logtail-rails/logger.rb

@@ -65,10 +65,15 @@ module Logtail
       options[:ingesting_port] ||= options[:telemetry_port] if options[:telemetry_port].present?
       options[:ingesting_scheme] ||= options[:telemetry_scheme] if options[:telemetry_scheme].present?
 
-      if ENV['LOGTAIL_SKIP_LOGS'].blank? && !Rails.env.test?
-        io_device = Logtail::LogDevices::HTTP.new(source_token, options)
-      else
+      if ENV['LOGTAIL_SKIP_LOGS'].present? || Rails.env.test?
         io_device = STDOUT
+      elsif source_token.blank?
+        # Don't break booting the app without the token, e.g. for `assets:precompile` in a Docker build
+        @blank_source_token = true
+        io_device = STDOUT
+      else
+        io_device = Logtail::LogDevices::HTTP.new(source_token, options)
+        @better_stack_logger_created = true
       end
 
       logger = self.create_logger(io_device)
@@ -87,5 +92,12 @@ module Logtail
 
       logger
     end
+
+    class << self
+      # What {create_default_logger} did, Logtail::Frameworks::Rails::Railtie warns about it once the app has booted
+      #
+      # @private
+      attr_reader :blank_source_token, :better_stack_logger_created
+    end
   end
 end

data/lib/logtail-rails/railtie.rb

@@ -43,6 +43,31 @@ module Logtail
             config.app_middleware.use middleware_class
           end
         end
+
+        # Warns about problems with the logger create_default_logger created, unless the integration is turned off.
+        # Registered here, so that it runs after the app's own after_initialize blocks, which may still broadcast to it.
+        initializer(:logtail_logger_check, after: :load_config_initializers) do |app|
+          app.config.after_initialize do
+            next unless Integrations::Rails.enabled?
+
+            if Logtail::Logger.blank_source_token
+              Kernel.warn("Logtail: the source token passed to Logtail::Logger.create_default_logger is blank, logging to STDOUT instead of sending logs to Better Stack.")
+              next
+            end
+            next unless Logtail::Logger.better_stack_logger_created
+
+            # A `config.logger = ...` line after config/application.rb, like the one generated in
+            # config/environments/production.rb, replaces the Better Stack logger without any error
+            loggers = ::Rails.logger.respond_to?(:broadcasts) ? ::Rails.logger.broadcasts : [::Rails.logger]
+            next if loggers.any? { |logger| logger.is_a?(Logtail::Logger) }
+
+            # Before Rails 7.1, broadcasting extends the logger with an anonymous module, which hides the target
+            extended_modules = ::Rails.logger.singleton_class.included_modules - ::Rails.logger.class.included_modules
+            next if extended_modules.any? { |mod| mod.name.nil? }
+
+            Kernel.warn("Logtail: Rails.logger isn't the Better Stack logger created by Logtail::Logger.create_default_logger, and doesn't broadcast to it, so your logs don't reach Better Stack. Most likely config.logger is set again later, e.g. in config/environments/production.rb.")
+          end
+        end
       end
     end
   end

data/lib/logtail-rails/version.rb

@@ -1,7 +1,7 @@
 module Logtail
   module Integrations
     module Rails
-      VERSION = "0.2.15"
+      VERSION = "0.3.1"
     end
   end
 end

data/lib/logtail-rails.rb

@@ -2,6 +2,10 @@ require "logtail-rails/overrides"
 
 require "logtail"
 
+# Rails' own files rely on Active Support being loaded first, and on Rails main (8.2.0.alpha) rails/railtie
+# fails without it: "uninitialized constant ActiveSupport::DescendantsTracker::WeakSet". It comes after logtail,
+# which loads Ruby's logger: older Active Support releases need that with concurrent-ruby 1.3.5 or newer.
+require "active_support"
 require "rails/railtie"
 require "active_record"
 require "rack"
@@ -50,6 +54,7 @@ module Logtail
         ActionView.integrate!
         ActiveRecord.integrate!
         RackLogger.integrate!
+        filter_parameters_in_urls(::Rails.application.config.filter_parameters)
       end
 
       def self.enabled=(value)
@@ -71,6 +76,16 @@ module Logtail
         @middlewares ||= [Logtail::Integrations::Rack::HTTPContext, SessionContext, Logtail::Integrations::Rack::UserContext,
           Logtail::Integrations::Rack::HTTPEvents, Logtail::Integrations::Rails::ErrorEvent].select(&:enabled?)
       end
+
+      # Filters the app's filter_parameters from the query strings and the Referer and Location
+      # URLs that Rack::HTTPEvents logs, unless the app customised its query_string_filters.
+      # Procs are left out: they rewrite values, while query string filters match names.
+      def self.filter_parameters_in_urls(filter_parameters)
+        http_events = Logtail::Integrations::Rack::HTTPEvents
+        return if http_events.query_string_filters != http_events::DEFAULT_QUERY_STRING_FILTERS
+
+        http_events.query_string_filters = filter_parameters.reject { |filter| filter.is_a?(Proc) }
+      end
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: logtail-rails
 version: !ruby/object:Gem::Version
-  version: 0.2.15
+  version: 0.3.1
 platform: ruby
 authors:
 - Better Stack
@@ -237,8 +237,16 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- ".github/actions/cleanup-sources/action.yml"
+- ".github/actions/cleanup-sources/run.sh"
+- ".github/actions/create-source/action.yml"
+- ".github/actions/create-source/run.sh"
+- ".github/actions/verify-logs/action.yml"
+- ".github/actions/verify-logs/run.sh"
+- ".github/workflows/e2e.yml"
 - ".github/workflows/main.yml"
 - ".github/workflows/no-closing-keywords.yml"
+- ".github/workflows/notify_linear_failure.yml"
 - ".github/workflows/release.yml"
 - ".gitignore"
 - ".rspec"
@@ -246,6 +254,10 @@ files:
 - LICENSE.md
 - README.md
 - Rakefile
+- e2e/example.sh
+- e2e/expected.json
+- e2e/installed.rb
+- e2e/plan.sh
 - example-project/.gitattributes
 - example-project/.gitignore
 - example-project/Gemfile
@@ -336,7 +348,6 @@ files:
 - gemfiles/rails-7.2.gemfile
 - gemfiles/rails-8.0.gemfile
 - gemfiles/rails-8.1.gemfile
-- gemfiles/rails-edge.gemfile
 - lib/logtail-rails.rb
 - lib/logtail-rails/action_controller.rb
 - lib/logtail-rails/action_controller/log_subscriber.rb

data/gemfiles/rails-edge.gemfile

@@ -1,13 +0,0 @@
-source 'https://rubygems.org'
-
-gem 'rails', github: 'rails/rails', branch: 'main'
-gem 'sidekiq', '>= 7.3.0', require: false if RUBY_VERSION >= '2.7.0'
-
-gem 'logtail'
-gem 'logtail-rack'
-
-if RUBY_PLATFORM == "java"
-  gem 'mime-types', '2.6.2'
-end
-
-gemspec :path => '../'