llm_cost_tracker 0.14.0 → 0.14.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ffc5fef0c3c18c6926b6c414d56fb0b7c9c4afef6db315139f8bc4626864611a
-  data.tar.gz: 3b012c1014c7246dd0eebb5413e13c02023cce056312e719a96328e86de749a8
+  metadata.gz: af338b0d20bfcbdfd9b95a88f903fa2a6b22dab16c72857a48bb2ff6ac0e9ac6
+  data.tar.gz: e2688d3cc3d29795787602ccf79b70dbe541347884644b3d5d234d1635f3f4b9
 SHA512:
-  metadata.gz: b28d7d695a1d6c45fe6c9d77d4f5caebbbde84682b7a261e25123d280181d00361baae80e9ddd65def0b9602c385d48813467ae741a5cde4bc378e4652a45829
-  data.tar.gz: 617cafb3be05250fd7c9c0b049a37474e1e0a2737d72cfd2744110cd531b68c362e7d15928f54d9a9c2d810da3f451516ff9ae747103bca2d618b77159359a13
+  metadata.gz: d4e06cd0bab7dd941fb958ca5fff5b6844ba17879f04bd3370258864e7ebd390088e091d2e0bd7f6fd839fcf05ecf3bc79f954b918c5b0b1cea471e1d14a413b
+  data.tar.gz: 107d59de744aabfbf946cbc38ce9ad705b4562a516446163dfb4834102ea263065d67e72c302c75f90ac887c12dfe11c404e85256e46300cd5a2ebe73b61b3af

data/CHANGELOG.md

@@ -2,6 +2,52 @@
 
 Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning: [SemVer](https://semver.org/spec/v2.0.0.html).
 
+## [0.14.1] - 2026-09-25
+
+### Added
+
+- `bin/rails llm_cost_tracker:doctor` and the boot log warn when an instrumented SDK is newer than its tested range (RubyLLM 3.0 and later), instead of the integration reporting installed while it records nothing.
+
+### Changed
+
+- Ruby 3.3 is supported; the minimum drops from 3.4.
+- The Faraday middleware adds `stream_options: { include_usage: true }` only for OpenAI, OpenRouter, DeepSeek, Groq, and Azure OpenAI on the v1 API or `api-version` 2024-06-01 and later without On Your Data or image input, since other servers can reject it and fail the request. Hosts you add to `config.capture.openai_compatible_providers` are left untouched: set the flag yourself if they support it, or their streams record `usage_source: unknown` with a warning.
+- `bin/rails llm_cost_tracker:prices:refresh` refuses a snapshot that zeroes an existing price or charges for a free one, drops a model's `input` or `output` rate, moves a price 100-fold or more, or switches currency, and leaves your pricing file untouched, so one bad commit to the snapshot can no longer silence budgets or make `:block_requests` block every call. `PREVIEW=1` and `prices:check` list the changes; re-run with `FORCE=1` (or `force: true`) to accept them. New models and smaller changes are not checked, so keep reviewing the refreshed file.
+- `pricing.unknown_model_behavior = :raise` records the call, with `cost_status: unknown`, before it raises `LlmCostTracker::UnknownPricingError`, so a call the provider already billed is no longer missing from the ledger.
+- A malformed `pricing.file` fails `LlmCostTracker.configure` at boot instead of every LLM call. A missing one is logged at boot and reported as an error by `doctor`, and calls use bundled prices until `prices:refresh` creates it; before, every call raised and nothing was recorded.
+
+### Removed
+
+- An OpenAI or Anthropic SDK stream that your code never iterates is no longer recorded at garbage collection; the finalizer that did it kept every stream alive until process exit.
+
+### Fixed
+
+- Per-tag `:block_requests` budgets check tags passed to the Faraday middleware (`f.use :llm_cost_tracker, tags: ...`) before the call is sent; they saw only `with_tags` tags, so an over-budget tenant still reached the provider.
+- RubyLLM transcriptions are priced at the model's audio input rate when it has one; they were priced as text, at a third to a half of the real cost.
+- `RubyLLM.transcribe` with a Gemini or Vertex AI model on RubyLLM 1.x is recorded; RubyLLM 1.x sent it through its own Gemini method, bypassing the wrapped `RubyLLM::Provider#transcribe`.
+- Calls made through RubyLLM 2.x are recorded. RubyLLM 2.0 moved token counts behind `response.tokens`, so every call was dropped with a warning while `doctor` reported the integration installed. A 2.x `paint(count:)` returning several images is recorded once, as RubyLLM bills it.
+- An automatically captured LLM call no longer fails, losing the provider's response, when the gem cannot record it, such as when the async inbox pool times out. Only `BudgetExceededError`, `UnknownPricingError` under `:raise`, and `TransactionAbortedError` reach your code; other recording failures are logged. `LlmCostTracker.track` and `track_stream` still raise them.
+- A recording error no longer replaces an exception your code raised while iterating an OpenAI or Anthropic SDK stream or inside `track_stream`, such as `Sidekiq::Shutdown`; the stream is still recorded, tagged `stream_errored`, and the recording error is logged. `TransactionAbortedError` still wins, with your exception as its cause.
+- An OpenAI or Anthropic batch with an unpriced model under `:raise`, or one crossing a budget, records every result before raising once; results after the failing one were lost and `batches.retrieve` raised on every poll.
+- With `ingestion.mode = :async`, one inbox row the database rejects no longer fails its whole batch, which after five attempts quarantined up to 99 other calls and dropped them from the ledger and budget totals. A failed batch is retried one row at a time and only rejected rows are marked failed. Rows already quarantined this way can be requeued as described in `docs/operations.md`.
+- A NUL byte in a stored string, which PostgreSQL rejects, or invalid UTF-8 in a tag value no longer loses the call, and `LlmCostTracker.with_tags` no longer raises on invalid UTF-8. NUL bytes are removed on write, including from inbox rows written by earlier releases, and invalid UTF-8 in tag values becomes U+FFFD.
+- An interrupted Gemini or Azure stream records its model from the request URL instead of `unknown`.
+- The async inbox's `last_error` is no longer empty when the 1,000-byte cut splits a multibyte character.
+- An LLM call inside your own database transaction no longer breaks it when the ledger write fails; on PostgreSQL your app's writes were lost with `PG::InFailedSqlTransaction`. Ledger writes, budget reads, and batch de-duplication run in a savepoint inside an open transaction. On MySQL, where a deadlock rolls back your whole transaction, the gem raises `LlmCostTracker::TransactionAbortedError` instead of swallowing it.
+- With `config.budgets.totals_source = :cache`, the rollup increment inside your transaction waits for its commit on Rails 7.2 and later, so an open transaction no longer holds the rollup row lock and stalls every request recording spend for the same provider. On Rails 7.1 and in non-joinable transactions such as transactional fixtures it still runs immediately, and it is never retried inside a transaction.
+- Long streams record their real usage instead of zero tokens and `$0`. Streams were buffered up to 1 MB, dropping the final usage event (past about 3,000 output tokens through Faraday); they are now decoded as they arrive, keeping only the first and last events plus billable tool-call and grounding events, so memory stays flat at any length. Image streams through Faraday keep their final usage too.
+- Streams captured through the OpenAI and Anthropic SDK integrations are released once your code drops them, instead of staying in memory with their buffered events and request for the life of the process.
+- OpenAI Realtime cached audio is no longer billed twice, at the cache-read rate and again at the audio rate ($0.258 instead of $0.106 for a 10,000-token `gpt-realtime` turn with 5,000 cached audio tokens). Cached audio and image tokens are priced at the model's cache-read rate, below OpenAI's published cached-audio price on the mini models. OpenAI-compatible usage whose audio and image buckets exceed the reported input has the overlap taken out of audio input, then image input.
+- Dashboard links and filter forms carry only the dashboard's own query parameters. They copied every parameter, and Rails reads some as link options, so a crafted link such as `?script_name=//evil.example` pointed Export CSV, pagination, filter, and tag links at another site, others rewrote links or caused a 500, and a 50 KB junk parameter grew a tag page to 10 MB. A query string over 16 KB is a bad request.
+- A dashboard page accepts at most 10 `tag[...]` filters, counting a tag value page's own value, and answers more with a bad request. Each filter adds a subquery, so PostgreSQL planning memory grew with the square of their count, and around 700 got the backend killed for running out of memory, restarting the database. Tag breakdowns no longer offer drill-down links past the limit.
+- Dashboard filters handle bad input without a 500. A list or hash in a tag, provider, model, stream, or usage-source filter is a bad request instead of silently matching nothing, and a NUL byte in a filter value matches nothing on PostgreSQL instead of raising. A CSV export with an invalid filter, a missing call, or a database error renders an HTML error page with the right status instead of a 500 or HTML labelled as CSV. A tag value page's value overrides a filter on the same key, and tag pages for keys containing a dot, such as `/tags/team.name`, load instead of returning 404 or 406.
+- The Pricing page shows the active price source instead of a 500 when `source` is a list or hash, as in `/pricing?source[]=bundled`; a list or hash `provider` filter is a bad request, as on other pages.
+
+### Security
+
+- From v0.9.0 through v0.14.0, a provider API key sent as a URL parameter could be stored in call tags. When a middleware after `f.use :llm_cost_tracker`, such as `f.response :raise_error`, raised on a streamed Gemini `?key=` or Azure `?api-key=` request, the `stream_interrupted_error` tag stored Faraday's error message with the full URL, shown on the dashboard, in CSV exports, and in notification payloads. The tag now holds only the error class, and the HTTP status goes into a new `stream_interrupted_status` tag. Rotate any key that may have been exposed, and clear stored messages with `UPDATE llm_cost_tracker_call_tags SET value = split_part(value, ': ', 1) WHERE key = 'stream_interrupted_error'` on PostgreSQL or ``UPDATE llm_cost_tracker_call_tags SET value = SUBSTRING_INDEX(value, ': ', 1) WHERE `key` = 'stream_interrupted_error'`` on MySQL. With async ingestion, run it after the inbox drains, and delete quarantined inbox rows whose `payload` contains `stream_interrupted_error`.
+- Provider keys, tokens, `key=`/`token=`/`sig=` URL parameters, URL user info, and `Authorization` headers inside a longer string are replaced with `[REDACTED]` in tag values, log lines, the async inbox's `last_error`, `TransactionAbortedError` messages, and the `source_url` that `llm_cost_tracker:prices:refresh` writes and prints; before, only a tag value that was entirely a key was caught.
+
 ## [0.14.0] - 2026-08-26
 
 ### Added

data/README.md

@@ -8,7 +8,7 @@ Every call your app makes through RubyLLM, the official OpenAI and Anthropic SDK
 
 Not Langfuse, Helicone, or LiteLLM. No prompts, no traces, no replay. Spend attribution only.
 
-Requires Ruby 3.4+, Rails 7.1+, PostgreSQL or MySQL.
+Requires Ruby 3.3+, Rails 7.1+, PostgreSQL or MySQL.
 
 <picture> <source media="(prefers-color-scheme: dark)" srcset="docs/dashboard-overview-dark.png"> <img alt="LLM Cost Tracker dashboard" src="docs/dashboard-overview-light.png"> </picture>
 

data/app/controllers/llm_cost_tracker/application_controller.rb

@@ -4,11 +4,14 @@ require "securerandom"
 
 module LlmCostTracker
   class ApplicationController < ActionController::Base
+    MAX_QUERY_BYTES = 16 * 1024
+
     layout "llm_cost_tracker/application"
 
     protect_from_forgery with: :exception
 
     before_action :set_dashboard_security_headers
+    before_action :reject_oversized_query
     before_action :ensure_current_schema
     before_action :assign_dashboard_date_range
 
@@ -22,6 +25,12 @@ module LlmCostTracker
 
     private
 
+    def reject_oversized_query
+      return if request.query_string.bytesize <= MAX_QUERY_BYTES
+
+      raise LlmCostTracker::InvalidFilterError, "query string exceeds #{MAX_QUERY_BYTES / 1024} KB"
+    end
+
     def ensure_current_schema
       drift = LlmCostTracker::Dashboard::SetupState.current
       return unless drift
@@ -40,16 +49,20 @@ module LlmCostTracker
     end
 
     def render_database_error(_error)
-      render "llm_cost_tracker/errors/database", status: :internal_server_error
+      render_error_page("database", :internal_server_error)
     end
 
     def render_invalid_filter(error)
       @error_message = error.message
-      render "llm_cost_tracker/errors/invalid_filter", status: :bad_request
+      render_error_page("invalid_filter", :bad_request)
     end
 
     def render_not_found
-      render "llm_cost_tracker/errors/not_found", status: :not_found
+      render_error_page("not_found", :not_found)
+    end
+
+    def render_error_page(name, status)
+      render "llm_cost_tracker/errors/#{name}", status: status, formats: :html, content_type: "text/html"
     end
 
     def set_dashboard_security_headers

data/app/controllers/llm_cost_tracker/pricing_controller.rb

@@ -4,10 +4,10 @@ module LlmCostTracker
   class PricingController < ApplicationController
     def index
       @overview = Dashboard::PricingOverview.call
-      requested = params[:source]&.to_sym
+      requested = params[:source].to_s.to_sym
       @active_source = @overview.fetch(:sources).key?(requested) ? requested : @overview.fetch(:effective_source)
       @source_data = @overview.fetch(:sources).fetch(@active_source)
-      @provider_filter = params[:provider].to_s.presence
+      @provider_filter = Dashboard::Params.scalar(params[:provider], :provider).presence
       @rows = @source_data.fetch(:rows)
       @rows = @rows.select { |row| row.provider == @provider_filter } if @provider_filter
       @providers = @source_data.fetch(:rows).map(&:provider).compact.uniq.sort

data/app/controllers/llm_cost_tracker/tags_controller.rb

@@ -7,19 +7,20 @@ module LlmCostTracker
     end
 
     def show
-      scope = Dashboard::Filter.call(params: params)
-      @value = params[:tag_value].to_s
+      @value = Dashboard::Params.scalar(params[:tag_value], :tag_value)
 
       if @value.empty?
         @sort = params[:sort].to_s
         @dir = params[:dir].to_s
-        @breakdown = Dashboard::TagBreakdown.call(scope: scope, key: params[:key], sort: @sort, direction: @dir)
+        @breakdown = Dashboard::TagBreakdown.call(
+          scope: Dashboard::Filter.call(params: params), key: params[:key], sort: @sort, direction: @dir
+        )
       else
         @key = LlmCostTracker::Tags::Key.validate!(
           params[:key],
           error_class: LlmCostTracker::InvalidFilterError
         )
-        value_scope = scope.by_tag(@key, @value)
+        value_scope = Dashboard::Filter.call(params: params, tags: { @key => @value })
         @value_total_cost = value_scope.sum(:total_cost).to_f
         @value_calls = value_scope.count
         @value_points = Dashboard::TimeSeries.call(scope: value_scope)

data/app/helpers/llm_cost_tracker/application_helper.rb

@@ -135,7 +135,7 @@ module LlmCostTracker
     end
 
     def current_query(overrides = {})
-      request.query_parameters.symbolize_keys.merge(overrides)
+      request.query_parameters.symbolize_keys.slice(*LlmCostTracker::Dashboard::Params::QUERY_KEYS).merge(overrides)
     end
 
     def calls_query_for_model(provider:, model:)

data/app/helpers/llm_cost_tracker/dashboard_query_helper.rb

@@ -16,6 +16,11 @@ module LlmCostTracker
       query
     end
 
+    def tag_drilldown_allowed?(key)
+      tags = LlmCostTracker::Dashboard::Params.tag_query(current_query[:tag])
+      tags.except(key.to_s).size < LlmCostTracker::Dashboard::Filter::MAX_TAG_FILTERS
+    end
+
     def hidden_query_fields(query, prefix: nil)
       safe_join(query.flat_map do |key, value|
         name = prefix ? "#{prefix}[#{key}]" : key.to_s

data/app/models/llm_cost_tracker/call.rb

@@ -45,7 +45,7 @@ module LlmCostTracker
       def already_recorded?(provider:, provider_response_id:)
         return false if provider_response_id.to_s.empty?
 
-        where(provider: provider, provider_response_id: provider_response_id).exists?
+        Ledger::Isolation.guard(self) { where(provider: provider, provider_response_id: provider_response_id).exists? }
       end
 
       def by_tag(key, value) = by_tags(key => value)

data/app/services/llm_cost_tracker/dashboard/filter.rb

@@ -5,20 +5,23 @@ require "date"
 module LlmCostTracker
   module Dashboard
     class Filter
+      MAX_TAG_FILTERS = 10
+      MATCH_NOTHING = Arel.sql("1 = 0")
       STREAM_FILTER_OPTIONS = [
         ["Streaming only", "yes"],
         ["Non-streaming only", "no"]
       ].freeze
 
       class << self
-        def call(scope: LlmCostTracker::Call.all, params: {})
-          new(scope: scope, params: params).relation
+        def call(scope: LlmCostTracker::Call.all, params: {}, tags: {})
+          new(scope: scope, params: params, tags: tags).relation
         end
       end
 
-      def initialize(scope:, params:)
+      def initialize(scope:, params:, tags: {})
         @scope = scope
         @params = LlmCostTracker::Dashboard::Params.to_hash(params).symbolize_keys
+        @extra_tags = tags
       end
 
       def relation
@@ -27,13 +30,13 @@ module LlmCostTracker
         filtered_scope = apply_exact_filter(filtered_scope, :provider)
         filtered_scope = apply_exact_filter(filtered_scope, :model)
         filtered_scope = apply_stream_filter(filtered_scope)
-        filtered_scope = apply_usage_source_filter(filtered_scope)
+        filtered_scope = apply_exact_filter(filtered_scope, :usage_source)
         apply_tag_filters(filtered_scope)
       end
 
       private
 
-      attr_reader :scope, :params
+      attr_reader :scope, :params, :extra_tags
 
       def apply_date_filters(relation)
         from_date = Dashboard::DateRange.parse(params, :from)
@@ -50,21 +53,25 @@ module LlmCostTracker
       end
 
       def apply_exact_filter(relation, key)
-        value = normalized_string(params[key])
+        value = normalized_string(params[key], key)
         return relation if value.nil?
 
-        relation.where(key => value)
+        matchable?([value]) ? relation.where(key => value) : relation.where(MATCH_NOTHING)
       end
 
       def apply_tag_filters(relation)
-        tags = tag_params
+        tags = tag_params.merge(extra_tags)
         return relation if tags.empty?
+        if tags.size > MAX_TAG_FILTERS
+          raise LlmCostTracker::InvalidFilterError,
+                "at most #{MAX_TAG_FILTERS} tag filters are allowed, got #{tags.size}"
+        end
 
-        relation.by_tags(tags)
+        matchable?(tags.values) ? relation.by_tags(tags) : relation.where(MATCH_NOTHING)
       end
 
       def apply_stream_filter(relation)
-        value = normalized_string(params[:stream])
+        value = normalized_string(params[:stream], :stream)
         return relation if value.nil?
 
         case value.downcase
@@ -74,28 +81,24 @@ module LlmCostTracker
         end
       end
 
-      def apply_usage_source_filter(relation)
-        value = normalized_string(params[:usage_source])
-        return relation if value.nil?
-
-        relation.where(usage_source: value)
-      end
-
       def tag_params
         tags = LlmCostTracker::Dashboard::Params.to_hash(params[:tag])
 
         tags.each_with_object({}) do |(key, value), normalized|
-          value = normalized_string(value)
+          value = normalized_string(value, "tag filter value")
           next if value.nil?
 
           normalized[LlmCostTracker::Tags::Key.validate!(key, error_class: LlmCostTracker::InvalidFilterError)] = value
         end
       end
 
-      def normalized_string(value)
-        return nil if value.nil?
+      def matchable?(values)
+        values.none? { |value| value.include?("\0") } ||
+          !LlmCostTracker::Ledger::Schema::Adapter.postgresql?(scope.connection)
+      end
 
-        value.to_s.strip.presence
+      def normalized_string(value, name)
+        LlmCostTracker::Dashboard::Params.scalar(value, name).strip.presence
       end
     end
   end

data/app/services/llm_cost_tracker/dashboard/params.rb

@@ -3,6 +3,8 @@
 module LlmCostTracker
   module Dashboard
     module Params
+      QUERY_KEYS = %i[from to provider model tag tag_value stream usage_source cost_status sort dir page per].freeze
+
       class << self
         def to_hash(value)
           return {} if value.nil?
@@ -17,6 +19,14 @@ module LlmCostTracker
           {}
         end
 
+        def scalar(value, name)
+          if value.is_a?(Hash) || value.is_a?(Array) || value.respond_to?(:to_unsafe_h)
+            raise InvalidFilterError, "#{name} must be a single value"
+          end
+
+          value.to_s
+        end
+
         def tag_query(value)
           to_hash(value).each_with_object({}) do |(key, tag_value), tags|
             key = key.to_s

data/app/views/llm_cost_tracker/tags/show.html.erb

@@ -107,6 +107,7 @@
           </tr>
         </thead>
         <tbody>
+          <% drilldown_allowed = tag_drilldown_allowed?(params[:key]) %>
           <% @breakdown.rows.each do |row| %>
             <tr>
               <td><code class="lct-code-id"><%= row.value %></code></td>
@@ -117,6 +118,8 @@
               <td class="lct-num">
                 <% if row.value == "(untagged)" %>
                   <span class="lct-num-muted">n/a</span>
+                <% elsif !drilldown_allowed %>
+                  <span class="lct-num-muted">tag filter limit</span>
                 <% else %>
                   <%= link_to "Trend", tag_path(params[:key], current_query.merge(tag_value: row.value)), class: "lct-page-link" %>
                   <%= link_to "Calls", calls_path(calls_query_for_tag(key: params[:key], value: row.value)), class: "lct-page-link" %>

data/config/routes.rb

@@ -4,7 +4,12 @@ LlmCostTracker::Engine.routes.draw do
   root "dashboard#index"
   resources :calls, only: %i[index show], constraints: { id: /\d+/ }, defaults: { format: :html }
   resources :models, only: :index
-  resources :tags, only: %i[index show], param: :key, format: false
+  resources :tags,
+            only: %i[index show],
+            param: :key,
+            format: false,
+            constraints: { key: %r{[^/]+} },
+            defaults: { format: :html }
   get "data_quality", to: "data_quality#index", as: :data_quality
   get "pricing", to: "pricing#index", as: :pricing
 

data/lib/llm_cost_tracker/budget/per_tag.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require_relative "../ledger/isolation"
 require_relative "../ledger/schema/adapter"
 require_relative "../ledger/tags/encoding"
 
@@ -63,9 +64,11 @@ module LlmCostTracker
 
         def spend_by_value(key, values, window, bucket)
           started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-          totals = LlmCostTracker::CallTag
-                   .where(key: key, value: values, TIME_COLUMN => window_range(window, bucket))
-                   .group(:value).sum(COST_COLUMN)
+          totals = Ledger::Isolation.guard(LlmCostTracker::CallTag) do
+            LlmCostTracker::CallTag
+              .where(key: key, value: values, TIME_COLUMN => window_range(window, bucket))
+              .group(:value).sum(COST_COLUMN)
+          end
           warn_slow_read(key, window, Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at)
           totals
         end
@@ -88,7 +91,8 @@ module LlmCostTracker
         end
 
         def columns?
-          (LlmCostTracker::CallTag.column_names & [COST_COLUMN, TIME_COLUMN]).size == 2
+          LlmCostTracker::CallTag.table_exists? &&
+            (LlmCostTracker::CallTag.column_names & [COST_COLUMN, TIME_COLUMN]).size == 2
         end
 
         private

data/lib/llm_cost_tracker/capture/event_window.rb

@@ -0,0 +1,99 @@
+# frozen_string_literal: true
+
+require_relative "sse"
+
+module LlmCostTracker
+  module Capture
+    class EventWindow
+      HEAD_EVENTS = 16
+      TAIL_EVENTS = 32
+      HEAVY_STRING_BYTES = 8 * 1024
+      IGNORED_PAYLOAD_KEYS = %w[b64_json partial_image_b64].freeze
+
+      def initialize(notable: nil)
+        @notable = notable
+        @head = []
+        @kept = []
+        @tail = []
+        @bytes = 0
+        @overflowed = false
+      end
+
+      def push(data, type: nil)
+        return if @overflowed
+
+        event = { event: type, data: strip_heavy_payload(data) }
+        size = approximate_bytesize(event)
+        if @head.size < HEAD_EVENTS
+          @head << [event, size]
+        else
+          @tail << [event, size]
+          settle(@tail.shift) if @tail.size > TAIL_EVENTS
+        end
+        @bytes += size
+        overflow! if @bytes > SSE::LIMIT_BYTES
+      rescue TypeError, SystemStackError
+        overflow!
+      end
+
+      def events
+        (@head + @kept + @tail).map(&:first)
+      end
+
+      def overflowed?
+        @overflowed
+      end
+
+      private
+
+      def settle(entry)
+        return @kept << entry if notable?(entry.first[:data])
+
+        @bytes -= entry.last
+      end
+
+      def notable?(data)
+        @notable&.call(data)
+      rescue StandardError
+        false
+      end
+
+      def overflow!
+        @overflowed = true
+        @head = []
+        @kept = []
+        @tail = []
+      end
+
+      def strip_heavy_payload(value)
+        case value
+        when Hash
+          value.each_with_object({}) do |(key, nested), out|
+            next if IGNORED_PAYLOAD_KEYS.include?(key.to_s)
+
+            out[key] = strip_heavy_payload(nested)
+          end
+        when Array
+          value.map { |nested| strip_heavy_payload(nested) }
+        when String
+          value.bytesize > HEAVY_STRING_BYTES ? "" : value
+        else
+          value
+        end
+      end
+
+      def approximate_bytesize(value)
+        case value
+        when Hash
+          value.sum { |key, nested| approximate_bytesize(key) + approximate_bytesize(nested) + 4 }
+        when Array
+          value.sum { |nested| approximate_bytesize(nested) + 2 }
+        when Numeric, true, false, nil
+          8
+        else
+          value.to_s.bytesize + 2
+        end
+      end
+    end
+  end
+end