ez_logs_agent 0.1.10 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2a7479b12ee7dd0814929516f0474d7e05e393410d86a1c6d56117c854945809
-  data.tar.gz: 54af5890799ca5614373dae0e66d1b422b3abab3dee7d9d0f95c7f23039ffdd7
+  metadata.gz: fbc336d9f93ad71ed33b7c76b07965e751eb6b197ae74a4e29a700c6498b2d26
+  data.tar.gz: 1429316f73489ca63f6925d6e5e6bb97a73e6911aefc3f832ec8be125fa9a74d
 SHA512:
-  metadata.gz: a68790e445b19ba92f2df2d0733fe6ae7e6d9c43f2ff854eed01fced091cb7d62aa1c0377429c44d3250abc515d5a1bb45056f1087eb0e2b8d491461fdd94843
-  data.tar.gz: f4208dc88f566f28e1ec24288533468fb3f99d268a77d4f3030da19fc4fc77fe8e36a39052516ddc94def977f03012d8bc157fccb319161a06d4889f52cd37e5
+  metadata.gz: 11b8ec5a85a5792c7d4ae0b90b01d46f9e738ba5e043df790b80f81db78da3e19b94611102acfd564388104852a4edf6b91d759f6a4310ffb405a4e97cb38ad0
+  data.tar.gz: acb11f65d81105396c16a5e189aa7437b607becbe46f8910c84828410546ec2ac19752af3f8a22414027623e375fbbdca52b4716c2670c97bd4119d432619ff3

data/CHANGELOG.md

@@ -2,6 +2,56 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.2.0] — 2026-06-05
+
+### Added
+- **Bulk database operations are now captured.** Adds a fourth event
+  `source_type` — `bulk_database` — for the four ActiveRecord operations
+  that bypass per-row callbacks: `delete_all`, `update_all`, `insert_all`,
+  `upsert_all`. Implemented via a narrowly-filtered
+  `ActiveSupport::Notifications.subscribe("sql.active_record")` subscription
+  (`Capturers::BulkDatabaseCapturer`) — not a replacement for the existing
+  callback-based DatabaseCapturer; the two run side by side under the
+  same `capture_database` config flag.
+
+  The new wire shape carries `model_class`, `operation`, `row_count`,
+  `where_template` + sanitized `where_binds`, plus an operation-specific
+  field (`set` for update_all, `columns` for insert_all/upsert_all).
+  Insert/upsert ship column NAMES only — no values, per the product
+  decision that bulk-row PII shouldn't ride the wire.
+
+  Cascade case (`dependent: :delete_all` on a parent destroy) is
+  captured automatically, since it produces the same SQL shape — the
+  reader sees the parent destroy AND the cascade as sibling rows on the
+  timeline.
+
+  Resource attribution uses a `resource_id: "bulk:<row_count>"` sentinel,
+  since individual row IDs are not knowable from the SQL without
+  changing the customer's operation (which would violate the read-only
+  principle).
+
+### Changed
+- `Sanitizer::SENSITIVE_PATTERNS` and the previous in-class
+  `DatabaseCapturer::SENSITIVE_PATTERNS` (which had drifted) are now a
+  single source of truth: `EzLogsAgent::SensitivePatterns::PATTERNS`.
+  Both capturers + the new BulkDatabaseCapturer consult the same list.
+  The merged list is the UNION of the previous two — no patterns
+  removed; `passwd`, `pwd`, `cvv`, `cvc`, `pem`, `cipher`, `nonce`,
+  `salt`, `digest`, `signature`, `hmac` all continue to be masked.
+- `encrypts :foo` introspection (Rails 7+ `model.class.encrypted_attributes`)
+  is now exposed as the standalone `EzLogsAgent::EncryptedAttributes`
+  module, so both the per-row and bulk capturers consult it via the
+  same path. Behavior unchanged for per-row.
+
+### Limits (documented)
+- Raw `connection.execute(sql)` calls are not captured (no
+  notifications fire under `sql.active_record` with a recognizable
+  shape). Use the typed bulk methods to get visibility.
+- Specific row IDs affected by a bulk op are not captured — only the
+  filter rule (WHERE columns + values) and row count. For per-row
+  detail, use `find_each(&:destroy)` style which fires per-row
+  callbacks.
+
 ## [0.1.10] — 2026-06-05
 
 ### Fixed

data/README.md

@@ -218,17 +218,29 @@ Sidekiq and ActiveJob executions:
 
 ActiveRecord create, update, and destroy operations:
 
-**What's captured:**
+**What's captured (per-row, via `after_create` / `after_update` / `after_destroy`):**
 - Model class name, record ID, operation type (create/update/destroy)
 - For creates: initial attribute values
 - For updates: one meaningful attribute change (e.g., `status: pending → shipped`)
 - For destroys: final attribute values before deletion
 - Correlation ID (automatically inherited from the current request or job)
 
+**What's captured (bulk, via `ActiveSupport::Notifications` on `sql.active_record`):**
+- The four ActiveRecord operations that bypass per-row callbacks:
+  `delete_all`, `update_all`, `insert_all`, `upsert_all`.
+- For each: model class, row count, the filter rule (WHERE columns +
+  sanitized bind values), and operation-specific detail (SET hash for
+  `update_all`, column names for `insert_all` / `upsert_all` — values
+  are NOT shipped for bulk inserts).
+- `dependent: :delete_all` cascades during a parent destroy.
+
 **What's NOT captured:**
 - SELECT queries (read operations don't change data)
 - Schema migrations (Rails internal operations)
-- Bulk operations (e.g., `update_all`, `delete_all`)
+- Raw `connection.execute(sql)` calls (no recognizable model class)
+- Individual row IDs affected by bulk operations (only the filter rule
+  and row count — pulling the IDs would require modifying the
+  customer's SQL, which violates the read-only principle).
 
 **Automatic exclusions (no configuration needed):**
 - `sessions` — Session store updates
@@ -777,18 +789,23 @@ Common validation errors:
 
 3. **Look for database capture registration in logs:**
    ```
-   [Railtie] Database capture installed
+   [Railtie] Database capture installed (per-row + bulk)
    ```
 
 4. **Verify models aren't excluded:**
    Check `config.excluded_tables` to ensure your tables aren't being filtered out.
 
-5. **Remember: Only create/update/destroy are captured:**
-   - ✅ `User.create(...)` — Captured
-   - ✅ `user.update(...)` — Captured
-   - ✅ `user.destroy` — Captured
+5. **What is and isn't captured:**
+   - ✅ `User.create(...)` — Captured (per-row callback)
+   - ✅ `user.update(...)` — Captured (per-row callback)
+   - ✅ `user.destroy` — Captured (per-row callback)
+   - ✅ `User.update_all(...)` — Captured as a `bulk_database` event
+     (filter + SET clause + row count; no per-row IDs)
+   - ✅ `User.where(...).delete_all` — Captured as a `bulk_database` event
+   - ✅ `User.insert_all(...)` / `upsert_all(...)` — Captured (columns + count)
    - ❌ `User.find(...)` — NOT captured (read-only)
-   - ❌ `User.update_all(...)` — NOT captured (bulk operation)
+   - ❌ `User.connection.execute("DELETE FROM ...")` — NOT captured
+     (raw SQL bypasses the typed AR API we hook)
 
 ---
 

data/lib/ez_logs_agent/bulk_sql_parser.rb

@@ -0,0 +1,302 @@
+# frozen_string_literal: true
+
+module EzLogsAgent
+  # Pure-functional parser for the four ActiveRecord bulk operations that
+  # bypass per-row callbacks: delete_all, update_all, insert_all, upsert_all.
+  # Used by BulkDatabaseCapturer to turn an AS::Notifications "sql.active_record"
+  # payload into a structured wire shape the server can humanize.
+  #
+  # ## What it extracts
+  #
+  # For delete_all:
+  #   { operation: :delete_all, where_template: "status = $1", where_binds: [{column:, value:}] }
+  #
+  # For update_all:
+  #   { operation: :update_all, set: {"status" => "paid"}, where_template:, where_binds: }
+  #
+  # For insert_all / upsert_all:
+  #   { operation: :insert_all|:upsert_all, columns: ["name", "email"] }
+  #   (No values — per the product decision; column SHAPE only.)
+  #
+  # For anything else (subqueries, joins, raw SQL the regex can't parse,
+  # malformed binds): returns { unparseable: true }. BulkDatabaseCapturer
+  # falls back to shipping row_count + operation + model_class with no
+  # template / binds / set — the timeline still reads "Bulk delete: 50,000
+  # orders" minus the WHERE detail.
+  #
+  # ## Why regex (not Arel / pg_query)
+  #
+  # Both Arel and pg_query are adapter-specific and either slow or heavy
+  # to add as a runtime dependency on every customer host app. Regex on
+  # the standardized AR-emitted SQL string is fast (sub-millisecond on
+  # typical statements) and adapter-tolerant. The graceful "unparseable"
+  # branch covers the long tail.
+  #
+  # ## Adapter quoting handled
+  #
+  # - PostgreSQL: "orders"."status" = $1   (double-quoted, $N placeholders)
+  # - SQLite:     "orders"."status" = ?     (double-quoted, ? placeholders)
+  # - MySQL:      `orders`.`status` = ?     (backticks, ? placeholders)
+  module BulkSqlParser
+    # Loose match for an identifier wrapped in any of the three quote
+    # styles AR uses. Captures the unquoted name.
+    IDENTIFIER = /["`]([^"`]+)["`]/.freeze
+
+    # Either a "qualified" or "bare" identifier reference, used in
+    # WHERE/SET clause columns. AR prefixes the table name in some
+    # paths and not in others — accept both. Captures just the column.
+    COLUMN = /(?:#{IDENTIFIER}\.)?#{IDENTIFIER}/.freeze
+
+    # A bind placeholder — Postgres uses $N (1-indexed), SQLite/MySQL use ?.
+    # The order of placeholders in the SQL corresponds 1:1 with the order
+    # of values in `binds` / `type_casted_binds`.
+    PLACEHOLDER = /(?:\$\d+|\?)/.freeze
+
+    module_function
+
+    # @param sql [String] Raw SQL string from `payload[:sql]`.
+    # @param type_casted_binds [Array] `payload[:type_casted_binds]` —
+    #   already-typecast values in placeholder order. May be nil for
+    #   raw SQL paths.
+    # @return [Hash] Always returns a Hash. Either the parsed structure
+    #   (keys above) or `{ unparseable: true }`.
+    def parse(sql:, type_casted_binds:)
+      return { unparseable: true } if sql.nil? || sql.empty?
+
+      sql_stripped = sql.strip
+
+      case sql_stripped
+      when /\ADELETE FROM /i
+        parse_delete(sql_stripped, type_casted_binds || [])
+      when /\AUPDATE /i
+        parse_update(sql_stripped, type_casted_binds || [])
+      when /\AINSERT INTO /i
+        parse_insert(sql_stripped)
+      else
+        { unparseable: true }
+      end
+    rescue StandardError
+      # If the regex engine, binds zipping, or any sub-parse step raises,
+      # ship unparseable rather than crash the capture handler. The
+      # BulkDatabaseCapturer's own rescue would also catch this, but
+      # defending here means the rest of the capturer sees a uniform
+      # return shape.
+      { unparseable: true }
+    end
+
+    # Returns the symbolic operation name we expect downstream. Detected
+    # from SQL shape, independent of the `payload[:name]` Rails version
+    # variance (see plan §"insert_all/upsert_all payload :name varies").
+    # Upsert detection requires the `ON CONFLICT` (PG) or `ON DUPLICATE KEY`
+    # (MySQL) clause — bare INSERTs are insert_all.
+    #
+    # @return [Symbol, nil] :delete_all, :update_all, :insert_all,
+    #   :upsert_all, or nil if not a bulk op.
+    def detect_operation(sql)
+      return nil if sql.nil?
+
+      sql_up = sql.lstrip.upcase
+
+      return :delete_all if sql_up.start_with?("DELETE FROM ")
+      return :update_all if sql_up.start_with?("UPDATE ")
+
+      if sql_up.start_with?("INSERT INTO ")
+        # Disambiguate insert_all vs upsert_all:
+      # - insert_all on PG/SQLite emits `ON CONFLICT DO NOTHING` (still insert).
+      # - upsert_all on PG/SQLite emits `ON CONFLICT ... DO UPDATE SET ...`.
+      # - upsert_all on MySQL emits `ON DUPLICATE KEY UPDATE ...`.
+        if (sql_up.include?("ON CONFLICT") && sql_up.include?("DO UPDATE")) ||
+           sql_up.include?("ON DUPLICATE KEY")
+          return :upsert_all
+        end
+
+        :insert_all
+      end
+    end
+
+    # --- DELETE FROM "orders" WHERE "orders"."status" = $1 ---
+    def parse_delete(sql, type_casted_binds)
+      where_sql = extract_where(sql)
+      template, binds = build_template_and_binds(where_sql, type_casted_binds)
+
+      { operation: :delete_all, where_template: template, where_binds: binds }
+    end
+
+    # --- UPDATE "orders" SET "status" = $1 WHERE "orders"."status" = $2 ---
+    # SET binds come first in placeholder order, then WHERE binds.
+    def parse_update(sql, type_casted_binds)
+      set_sql = extract_set(sql)
+      where_sql = extract_where(sql)
+      return { unparseable: true } if set_sql.nil?
+
+      set_pairs, set_bind_count = parse_set_assignments(set_sql)
+      set_binds = type_casted_binds.first(set_bind_count)
+      where_binds_raw = type_casted_binds.drop(set_bind_count)
+
+      set_hash = zip_set_values(set_pairs, set_binds)
+      where_template, where_binds = build_template_and_binds(where_sql, where_binds_raw)
+
+      {
+        operation: :update_all,
+        set: set_hash,
+        where_template: where_template,
+        where_binds: where_binds
+      }
+    end
+
+    # --- INSERT INTO "users" ("name","email") VALUES (...), (...) ---
+    def parse_insert(sql)
+      operation = detect_operation(sql)
+      return { unparseable: true } unless operation == :insert_all || operation == :upsert_all
+
+      columns = extract_insert_columns(sql)
+      return { unparseable: true } if columns.nil? || columns.empty?
+
+      { operation: operation, columns: columns }
+    end
+
+    # Returns the WHERE-clause SQL (without the keyword), or nil if absent.
+    # Stops at end-of-string, RETURNING, ORDER BY, LIMIT (AR rarely emits
+    # these on bulk DML but cheap insurance).
+    def extract_where(sql)
+      match = sql.match(/\sWHERE\s+(.+?)(?:\s+(?:RETURNING|ORDER BY|LIMIT)\b|\z)/i)
+      match && match[1].strip
+    end
+
+    # Returns the SET-clause SQL between "SET" and "WHERE"/end-of-string.
+    def extract_set(sql)
+      match = sql.match(/\sSET\s+(.+?)(?:\s+WHERE\s+|\s+RETURNING\b|\z)/i)
+      match && match[1].strip
+    end
+
+    # SET clause is comma-separated `"col" = <placeholder|literal>` pairs.
+    # AR inlines non-symbol values as literals (no placeholder) — we still
+    # surface those in the result so the reader sees what changed.
+    # Returns [[col_name, placeholder_or_literal_str], ...] + total bind count.
+    def parse_set_assignments(set_sql)
+      pairs = []
+      bind_count = 0
+
+      split_top_level_commas(set_sql).each do |assignment|
+        m = assignment.match(/\A#{COLUMN}\s*=\s*(.+)\z/)
+        next unless m
+
+        col = m[2] || m[1]
+        rhs = m[3].strip
+        pairs << [col, rhs]
+        bind_count += 1 if rhs.match?(PLACEHOLDER)
+      end
+
+      [pairs, bind_count]
+    end
+
+    # Walks set_pairs in order, taking the next bind for each placeholder
+    # RHS and pulling the literal text for non-placeholder RHS (e.g.,
+    # `updated_at = '2026-06-05 12:00:00'`).
+    def zip_set_values(set_pairs, set_binds)
+      bind_idx = 0
+      set_pairs.each_with_object({}) do |(col, rhs), acc|
+        if rhs.match?(PLACEHOLDER)
+          acc[col] = set_binds[bind_idx]
+          bind_idx += 1
+        else
+          # Strip surrounding quotes to surface the actual value.
+          acc[col] = rhs.gsub(/\A['"]|['"]\z/, "")
+        end
+      end
+    end
+
+    # Splits on commas that are NOT inside (parens) or quotes. Bulk DML
+    # SET / WHERE almost never nests, but defensive against function
+    # calls like `coalesce(x, 0)`.
+    def split_top_level_commas(str)
+      result = []
+      depth = 0
+      in_quote = false
+      quote_char = nil
+      buffer = +""
+
+      str.each_char do |ch|
+        if in_quote
+          buffer << ch
+          in_quote = false if ch == quote_char
+        elsif ch == "'" || ch == '"' || ch == "`"
+          in_quote = true
+          quote_char = ch
+          buffer << ch
+        elsif ch == "("
+          depth += 1
+          buffer << ch
+        elsif ch == ")"
+          depth -= 1
+          buffer << ch
+        elsif ch == "," && depth.zero?
+          result << buffer.strip unless buffer.empty?
+          buffer = +""
+        else
+          buffer << ch
+        end
+      end
+      result << buffer.strip unless buffer.empty?
+      result
+    end
+
+    # Given a WHERE clause SQL and the binds in placeholder order, walk
+    # the placeholders and pair each one with the column to its LEFT
+    # (the standard `"table"."col" = $1` shape AR emits). Returns the
+    # template (placeholder-preserved) and the binds tagged with column.
+    #
+    # Unrecognized shapes (subselects, joins, NULL checks without binds)
+    # leave the template intact but produce no column for the bind, in
+    # which case the bind ships with column: nil. The display layer
+    # handles nil-column binds.
+    def build_template_and_binds(where_sql, type_casted_binds)
+      return [nil, []] if where_sql.nil?
+
+      template = where_sql
+      binds = []
+      bind_index = 0
+
+      # Walk the template scanning each placeholder, looking backward
+      # for the nearest column identifier to its left.
+      template.scan(PLACEHOLDER) do
+        match_data = Regexp.last_match
+        next unless match_data
+
+        column = nearest_column_left_of(template, match_data.begin(0))
+        value = type_casted_binds[bind_index]
+        binds << { column: column, value: value }
+        bind_index += 1
+      end
+
+      [template, binds]
+    end
+
+    # Finds the most recent column identifier to the left of `pos` in
+    # the WHERE clause. Used to attribute each bind to its column name
+    # so we can mask sensitive values by name downstream.
+    def nearest_column_left_of(where_sql, pos)
+      prefix = where_sql[0...pos]
+      # Look for the last identifier before the operator (=, <, >, etc.)
+      match = prefix.match(/#{COLUMN}\s*(?:=|<>|!=|<=|>=|<|>|LIKE|IN|IS)\s*\z/i)
+      return nil unless match
+
+      # COLUMN has two capture groups: [table, col] or [_, col].
+      match[2] || match[1]
+    end
+
+    # --- INSERT INTO "users" ("name","email") VALUES ...  ---
+    # Extracts the ordered column list. Returns nil if the open paren
+    # column list isn't present (e.g., INSERT ... DEFAULT VALUES).
+    def extract_insert_columns(sql)
+      # First parenthesized list after the table name.
+      m = sql.match(/\AINSERT INTO\s+#{IDENTIFIER}\s*\(([^)]+)\)/i)
+      return nil unless m
+
+      columns_block = m[2]
+      columns_block.split(",").map do |col|
+        col.strip.gsub(/\A["`]|["`]\z/, "")
+      end.reject(&:empty?)
+    end
+  end
+end

data/lib/ez_logs_agent/capturers/bulk_database_capturer.rb

@@ -0,0 +1,368 @@
+# frozen_string_literal: true
+
+require "active_support/notifications"
+
+module EzLogsAgent
+  module Capturers
+    # Captures bulk SQL operations that bypass ActiveRecord lifecycle
+    # callbacks: delete_all, update_all, insert_all, upsert_all.
+    #
+    # ## Why this exists
+    #
+    # DatabaseCapturer (the per-row sibling) hooks after_create/_update/_destroy
+    # to capture rich per-record context (saved_changes, encrypted_attributes,
+    # display_name). That model breaks for bulk ops — Rails issues a single
+    # UPDATE/DELETE/INSERT statement against the database WITHOUT instantiating
+    # records, so the callbacks never fire. Customer code like
+    #
+    #     Order.where(status: "cart").delete_all
+    #     Library.where(closed: true).update_all(status: "active")
+    #     User.insert_all([{name: "a"}, {name: "b"}])
+    #
+    # plus `dependent: :delete_all` cascades during a parent destroy, are all
+    # invisible to the callback-based path. This capturer fills the gap.
+    #
+    # ## How it works
+    #
+    # Subscribes to "sql.active_record" — the standard Rails instrumentation
+    # API every observability tool uses (Datadog APM, AppSignal, Skylight).
+    # On every SQL statement the host app runs, we get a payload with
+    # the raw SQL, binds, name, and row_count. We filter aggressively
+    # to ONLY four operations (delete_all / update_all / insert_all /
+    # upsert_all) by SQL shape detection (BulkSqlParser.detect_operation),
+    # then parse + sanitize + ship.
+    #
+    # ## Dedup vs DatabaseCapturer
+    #
+    # Per-row CRUD (`user.save`, `order.destroy`) fires `after_*` callbacks
+    # AND produces an `sql.active_record` notification with a singular name
+    # ("User Update", "Order Destroy"). DatabaseCapturer captures these via
+    # callbacks; this capturer ignores them because their SQL shape is NOT
+    # one of the four bulk operations. Mutually exclusive — no double-capture.
+    #
+    # Cascade case: `Company has_many :orders, dependent: :delete_all` issues
+    # a single DELETE for the children. Callbacks don't fire on the children
+    # (delete_all bypasses them by design), but this capturer catches the
+    # bulk DELETE. The parent's `after_destroy` is captured separately by
+    # DatabaseCapturer. Both events share the request's correlation_id and
+    # land under the same Action shell. Reader sees parent + cascade as
+    # sibling rows on the timeline — the right narrative.
+    #
+    # ## Wire shape (matches server EventIngest expectations)
+    #
+    #   {
+    #     source_type: "bulk_database",
+    #     source_data: {
+    #       model_class: "Order",
+    #       operation: "delete_all" | "update_all" | "insert_all" | "upsert_all",
+    #       row_count: 50000,
+    #       where_template: "\"orders\".\"status\" = $1",
+    #       where_binds: [{column: "status", value: "cart"}],
+    #       set: {"status" => "paid"},        # only update_all
+    #       columns: ["name", "email"]        # only insert_all / upsert_all
+    #     },
+    #     correlation_id: ...,
+    #     resource_ids: [{resource_type: "Order", resource_id: "bulk:50000"}],
+    #     outcome: "success",
+    #     duration_ms: <finish - start>
+    #   }
+    #
+    # The "bulk:<count>" sentinel resource_id is required because the server's
+    # ResourceAggregationStage drops entries with nil resource_id. The
+    # display layer detects the sentinel and renders "Order (50,000 rows)"
+    # without a clickable entity link.
+    module BulkDatabaseCapturer
+      # AR's `payload[:name]` convention for the four bulk operations
+      # (verified against Rails 7.0–8.0 + SQLite/PG/MySQL):
+      #
+      #   delete_all  → "<Model> Delete All"
+      #   update_all  → "<Model> Update All"
+      #   insert_all  → "<Model> Insert"   (or "<Model> Bulk Insert" on older PG)
+      #   upsert_all  → "<Model> Upsert"   (or "<Model> Bulk Upsert" on older PG)
+      #
+      # Per-row CRUD uses singular operation verbs:
+      #   user.save (new)     → "<Model> Create"
+      #   user.update         → "<Model> Update"  (no " All")
+      #   user.destroy        → "<Model> Destroy"
+      #
+      # So the four bulk shapes are uniquely identified by either:
+      #   - ending in " All" (covers Delete All / Update All), OR
+      #   - the words Insert / Upsert (which are NEVER used for per-row CRUD
+      #     — per-row inserts are tagged "Create", per-row updates "Update").
+      #
+      # SQL shape detection (BulkSqlParser.detect_operation) is the actual
+      # authority — this filter is only a sub-µs pre-pass to skip non-bulk
+      # notifications without parsing SQL.
+      BULK_NAME_HINT = / All\z| (Bulk )?(Insert|Upsert)\z/.freeze
+
+      class << self
+        attr_reader :subscriber
+
+        # Installs the AS::Notifications subscription. Idempotent — calling
+        # twice is a no-op (would otherwise produce double-events because
+        # AS::Notifications.subscribe is itself NOT idempotent).
+        #
+        # Called from Railtie.install_database_capturer alongside the
+        # per-row DatabaseCapturer.install. Both gated by the same
+        # `capture_database` configuration flag — no new toggle.
+        def install
+          return if @installed
+
+          @subscriber = ::ActiveSupport::Notifications.subscribe("sql.active_record") do |*args|
+            payload = args.last
+            event_name = args.first
+            started = args[1]
+            finished = args[2]
+            handle_notification(event_name, started, finished, payload)
+          end
+          @installed = true
+        end
+
+        # Removes the subscription. Specs use this between examples to
+        # avoid leaked subscribers; production never calls it.
+        def uninstall!
+          ::ActiveSupport::Notifications.unsubscribe(@subscriber) if @subscriber
+          @subscriber = nil
+          @installed = false
+        end
+
+        # Per-notification entry point. Wraps everything in `rescue Exception`
+        # because an AS::N handler that raises pollutes the host's subscriber
+        # chain and (depending on the chain order) can break OTHER observability
+        # tools listening on the same channel. Hard rule: bulk capture failures
+        # never propagate.
+        def handle_notification(_event_name, started, finished, payload)
+          return unless capture_enabled?
+          return unless eligible_payload?(payload)
+
+          operation = ::EzLogsAgent::BulkSqlParser.detect_operation(payload[:sql])
+          return unless operation
+
+          model_class = resolve_model_class(payload[:sql])
+          return if model_class.nil?
+          return if table_excluded?(model_class)
+
+          parse_result = ::EzLogsAgent::BulkSqlParser.parse(
+            sql: payload[:sql],
+            type_casted_binds: payload[:type_casted_binds]
+          )
+
+          source_data = build_source_data(
+            operation: operation,
+            model_class: model_class,
+            row_count: payload[:row_count],
+            parse_result: parse_result
+          )
+
+          duration_ms = ((finished - started) * 1000).to_i
+
+          event = ::EzLogsAgent::EventBuilder.build(
+            source_type: :bulk_database,
+            source_data: source_data,
+            outcome: :success,
+            correlation_id: ::EzLogsAgent::Correlation.current,
+            resource_ids: build_resource_ids(model_class, source_data[:row_count]),
+            context: nil,
+            duration_ms: duration_ms
+          )
+
+          ::EzLogsAgent::Buffer.push(event)
+        rescue Exception => e # rubocop:disable Lint/RescueException
+          # See class comment: a raise from an AS::N handler hurts other
+          # subscribers, so we swallow EVERYTHING (not just StandardError).
+          # Logged at error level so a regression surfaces in customer
+          # debug output, but never re-raised.
+          safe_log_error("handle_notification", e)
+        end
+
+        # Fast pre-filter — checks the name field WITHOUT touching SQL.
+        # Returns false for the vast majority of notifications (per-row
+        # CRUD, SCHEMA, TRANSACTION, internal lookups).
+        #
+        # @param payload [Hash, nil]
+        # @return [Boolean]
+        def eligible_payload?(payload)
+          return false unless payload.is_a?(Hash)
+
+          name = payload[:name].to_s
+          return false if name.empty?
+
+          BULK_NAME_HINT.match?(name)
+        end
+
+        # Looks up the model class from the SQL's table name. Returns nil
+        # for SQL we can't attribute (raw multi-table queries, anonymous
+        # adapter SQL, schema introspection). Skipping these is correct —
+        # we'd have nothing to display anyway.
+        def resolve_model_class(sql)
+          return nil if sql.nil?
+
+          table = extract_table_name(sql)
+          return nil if table.nil?
+
+          ::ActiveRecord::Base.descendants.find do |klass|
+            klass.respond_to?(:table_name) && klass.table_name == table && !klass.abstract_class?
+          end
+        rescue StandardError
+          nil
+        end
+
+        # Extracts the unquoted table name from the FROM / INTO / UPDATE
+        # clause. Handles all three identifier-quote styles (PG/SQLite/
+        # MySQL). Returns nil on unparseable SQL.
+        def extract_table_name(sql)
+          # DELETE FROM "table"
+          if (m = sql.match(/\ADELETE FROM\s+["`]?([^"`\s]+)["`]?/i))
+            return m[1]
+          end
+          # UPDATE "table"
+          if (m = sql.match(/\AUPDATE\s+["`]?([^"`\s]+)["`]?/i))
+            return m[1]
+          end
+          # INSERT INTO "table"
+          if (m = sql.match(/\AINSERT INTO\s+["`]?([^"`\s]+)["`]?/i))
+            return m[1]
+          end
+
+          nil
+        end
+
+        # Builds the source_data hash from the parser result, applying
+        # encrypted_attributes drop + sensitive-pattern masking on
+        # column-keyed values.
+        def build_source_data(operation:, model_class:, row_count:, parse_result:)
+          base = {
+            model_class: model_class.name,
+            operation: operation.to_s,
+            row_count: row_count
+          }
+
+          return base if parse_result[:unparseable]
+
+          if (set = parse_result[:set])
+            base[:set] = mask_set_hash(set, model_class)
+          end
+
+          if (template = parse_result[:where_template])
+            base[:where_template] = template
+            base[:where_binds] = mask_where_binds(parse_result[:where_binds], model_class)
+          end
+
+          if (columns = parse_result[:columns])
+            base[:columns] = filter_columns(columns, model_class)
+          end
+
+          base
+        end
+
+        # Walks `{ column => value }` from update_all SET, masking values
+        # whose column is encrypted OR matches a sensitive pattern. Date /
+        # Time / BigDecimal values get JSON-formatted so they don't collapse
+        # to "[Object]" downstream.
+        def mask_set_hash(set, model_class)
+          set.each_with_object({}) do |(col, value), acc|
+            acc[col] =
+              if ::EzLogsAgent::EncryptedAttributes.attribute?(model_class, col)
+                "[FILTERED]"
+              elsif ::EzLogsAgent::SensitivePatterns.match?(col)
+                "[FILTERED]"
+              else
+                format_value_for_json(value)
+              end
+          end
+        end
+
+        # Walks the array of {column:, value:} bind entries from the WHERE
+        # parser, same masking rules as mask_set_hash. Binds whose column
+        # is nil (the parser couldn't attribute them) ride through with
+        # the formatted value — display falls back to template substitution.
+        def mask_where_binds(binds, model_class)
+          (binds || []).map do |bind|
+            col = bind[:column]
+            value = bind[:value]
+            masked_value =
+              if col && (::EzLogsAgent::EncryptedAttributes.attribute?(model_class, col) ||
+                         ::EzLogsAgent::SensitivePatterns.match?(col))
+                "[FILTERED]"
+              else
+                format_value_for_json(value)
+              end
+
+            { column: col, value: masked_value }
+          end
+        end
+
+        # For insert_all / upsert_all, we ship column names ONLY (no
+        # values — product decision). Sensitive column names still need
+        # masking so the column LIST itself doesn't hint "this table has
+        # a `password` column". Drop sensitive columns from the displayed
+        # list; replace with the literal marker so the count remains true.
+        def filter_columns(columns, model_class)
+          columns.map do |col|
+            if ::EzLogsAgent::EncryptedAttributes.attribute?(model_class, col)
+              "[FILTERED]"
+            elsif ::EzLogsAgent::SensitivePatterns.match?(col)
+              "[FILTERED]"
+            else
+              col
+            end
+          end
+        end
+
+        # Builds the sentinel resource entry. row_count may be nil (Rails
+        # < 7 didn't ship it; some adapters still don't) — fall back to
+        # "bulk" so the entry is non-nil and the server-side
+        # ResourceAggregationStage doesn't drop it.
+        def build_resource_ids(model_class, row_count)
+          count_str = row_count.is_a?(Integer) ? row_count.to_s : "unknown"
+          [{ resource_type: model_class.name, resource_id: "bulk:#{count_str}" }]
+        end
+
+        # Mirrors DatabaseCapturer's same-named guard. capture_database = false
+        # disables both capturers in one switch.
+        def capture_enabled?
+          ::EzLogsAgent.configuration.capture_database
+        rescue StandardError
+          false
+        end
+
+        # Uses DatabaseCapturer's existing all_excluded_tables list — one
+        # config knob, both capturers obey it.
+        def table_excluded?(model_class)
+          return false unless model_class.respond_to?(:table_name)
+
+          ::EzLogsAgent.configuration.all_excluded_tables.include?(model_class.table_name)
+        rescue StandardError
+          false
+        end
+
+        # Same formatter as DatabaseCapturer. Keeps Date / Time / BigDecimal
+        # from collapsing to "[Object]" when they reach Sanitizer / wire.
+        def format_value_for_json(value)
+          case value
+          when ::Time, ::DateTime
+            value.iso8601
+          when ::Date
+            value.to_s
+          when ::BigDecimal
+            value.to_f
+          when ::Array
+            value.map { |v| format_value_for_json(v) }
+          else
+            value
+          end
+        end
+
+        def safe_log_error(stage, exception)
+          ::EzLogsAgent::Logger.error(
+            "[BulkDatabaseCapturer] #{stage} failed: #{exception.class} - #{exception.message}"
+          )
+        rescue StandardError
+          # Even logging can fail in pathological boot states. We've done
+          # everything reasonable; drop the event silently.
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/ez_logs_agent/capturers/database_capturer.rb

@@ -60,37 +60,9 @@ module EzLogsAgent
       # Previously we filtered them out, but this loses important context.
       # FOREIGN_KEY_PATTERN = /_id\z/  # Removed January 2026
 
-      # Patterns for sensitive data to ignore.
-      #
-      # The first source of truth is `record.class.encrypted_attributes`
-      # (Rails 7+ `encrypts :foo` declaration) — see encrypted_attribute?.
-      # If the host app encrypted it, we never capture it.
-      #
-      # This list is the secondary defense: column names that frequently
-      # carry sensitive material even when the host app didn't declare
-      # `encrypts` (legacy code, manual hashing, externally-generated
-      # material). Matching is substring + case-insensitive.
-      SENSITIVE_PATTERNS = %w[
-        password
-        token
-        secret
-        api_key
-        credit_card
-        ssn
-        social_security
-        encrypted
-        private_key
-        public_key
-        signing_key
-        pem
-        cipher
-        nonce
-        salt
-        digest
-        signature
-        hmac
-      ].freeze
-
+      # Sensitive-attribute name pattern denylist (secondary defense after
+      # `encrypts :foo` introspection) lives in EzLogsAgent::SensitivePatterns —
+      # see sensitive_attribute? below.
 
       @installed = false
       @callbacks_registered = false
@@ -386,35 +358,25 @@ module EzLogsAgent
         end
 
         # Checks whether the host app declared `encrypts :<attribute>` on
-        # this model's class. Available since Rails 7.0 via
-        # ActiveRecord::Encryption::EncryptableRecord#encrypted_attributes.
-        #
-        # Safe across host Rails versions: returns false if the API isn't
-        # present (older Rails, non-AR records).
+        # this model's class. Delegates to EncryptedAttributes (single
+        # source of truth shared with BulkDatabaseCapturer, which only has
+        # the class — no instance — for bulk operations).
         #
         # @param attribute [String] The attribute name (already to_s'd)
         # @param model [ActiveRecord::Base] The model instance
         # @return [Boolean]
         def encrypted_attribute?(attribute, model)
-          klass = model.class
-          return false unless klass.respond_to?(:encrypted_attributes)
-
-          encrypted = klass.encrypted_attributes
-          return false if encrypted.nil? || encrypted.empty?
-
-          encrypted.map(&:to_s).include?(attribute)
-        rescue StandardError
-          false
+          EzLogsAgent::EncryptedAttributes.attribute?(model.class, attribute)
         end
 
         # Checks if attribute name contains sensitive patterns.
-        # Secondary check — see SENSITIVE_PATTERNS comment.
+        # Delegates to SensitivePatterns (single source of truth shared
+        # with Sanitizer and BulkDatabaseCapturer).
         #
         # @param attribute [String] The attribute name
         # @return [Boolean]
         def sensitive_attribute?(attribute)
-          attr_lower = attribute.downcase
-          SENSITIVE_PATTERNS.any? { |pattern| attr_lower.include?(pattern) }
+          EzLogsAgent::SensitivePatterns.match?(attribute)
         end
 
         # Checks if both values are scalar types

data/lib/ez_logs_agent/encrypted_attributes.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module EzLogsAgent
+  # Primary defense against capturing encrypted columns: read the host
+  # app's `encrypts :foo` declarations (Rails 7+ ActiveRecord::Encryption)
+  # and drop those attributes from anywhere we'd ship them on the wire.
+  #
+  # Two callers:
+  # - DatabaseCapturer (per-record callbacks) — has a model INSTANCE,
+  #   used to also work from `record.class`.
+  # - BulkDatabaseCapturer (AS::Notifications path) — has only the model
+  #   CLASS (the bulk SQL never instantiated a record). So this module
+  #   takes a class, not an instance — both call sites converge.
+  #
+  # The Rails API is `ModelClass.encrypted_attributes` (Symbol array).
+  # Available since Rails 7.0; older Rails or non-AR classes return false
+  # from this module, which is the fail-open default for the encrypts
+  # check. The pattern-based fallback in SensitivePatterns is the second
+  # layer of defense for hosts that don't (or can't) declare encrypts.
+  module EncryptedAttributes
+    module_function
+
+    # @param model_class [Class, nil] The AR class
+    # @param attribute [String, Symbol, nil] The attribute name
+    # @return [Boolean] true iff the host app declared `encrypts :<attribute>`
+    #   on `model_class` (or an ancestor). False on any error, missing API,
+    #   or empty list — see comment about fail-open semantics above.
+    def attribute?(model_class, attribute)
+      return false if model_class.nil? || attribute.nil?
+      return false unless model_class.respond_to?(:encrypted_attributes)
+
+      declared = model_class.encrypted_attributes
+      return false if declared.nil? || declared.empty?
+
+      attribute_str = attribute.to_s
+      declared.any? { |declared_attr| declared_attr.to_s == attribute_str }
+    rescue StandardError
+      # Same rescue policy as the previous inline check in DatabaseCapturer:
+      # if introspection raises (host app monkey-patched the API, weird
+      # AR class hierarchy), fall through to the pattern-based fallback
+      # rather than crash the capture path.
+      false
+    end
+  end
+end

data/lib/ez_logs_agent/event_builder.rb

@@ -24,7 +24,10 @@ module EzLogsAgent
     SENSITIVE_KEYS = %w[password token secret api_key credit_card].freeze
 
     # Valid source types
-    VALID_SOURCE_TYPES = %w[http_request background_job database_callback].freeze
+    # bulk_database covers AR bulk ops that bypass per-row callbacks
+    # (delete_all, update_all, insert_all, upsert_all) captured via
+    # ActiveSupport::Notifications. See Capturers::BulkDatabaseCapturer.
+    VALID_SOURCE_TYPES = %w[http_request background_job database_callback bulk_database].freeze
 
     # Valid outcome values
     VALID_OUTCOMES = %w[success failure].freeze

data/lib/ez_logs_agent/railtie.rb

@@ -218,10 +218,13 @@ module EzLogsAgent
       EzLogsAgent::Logger.error("[Railtie] Failed to install ActiveJob capturer: #{e.class} - #{e.message}")
     end
 
-    # Install Database capturer
+    # Install Database capturers (per-row + bulk).
     #
-    # Database capturer installs ActiveRecord lifecycle callbacks
-    # (after_create, after_update, after_destroy) for all models.
+    # Two capturers, one switch (`capture_database`):
+    # - DatabaseCapturer: per-row CRUD via after_create / _update / _destroy.
+    # - BulkDatabaseCapturer: bulk SQL via ActiveSupport::Notifications
+    #   ("sql.active_record"), narrowly filtered to delete_all / update_all /
+    #   insert_all / upsert_all. Catches what callbacks can't see.
     #
     # @return [void]
     def self.install_database_capturer
@@ -240,8 +243,9 @@ module EzLogsAgent
       return if @database_capturer_installed
 
       EzLogsAgent::Capturers::DatabaseCapturer.install
+      EzLogsAgent::Capturers::BulkDatabaseCapturer.install
       @database_capturer_installed = true
-      EzLogsAgent::Logger.debug("[Railtie] Database capture installed")
+      EzLogsAgent::Logger.debug("[Railtie] Database capture installed (per-row + bulk)")
     rescue StandardError => e
       EzLogsAgent::Logger.error("[Railtie] Failed to install database capturer: #{e.class} - #{e.message}")
     end

data/lib/ez_logs_agent/sanitizer.rb

@@ -21,18 +21,11 @@ module EzLogsAgent
   # The module is pure (no I/O, no state), so it's safe to call from
   # any thread.
   module Sanitizer
-    # Default sensitive-key patterns. Matched case-insensitively as
-    # SUBSTRINGS of the key, so `customer_password` matches `password`.
-    SENSITIVE_PATTERNS = %w[
-      password passwd pwd
-      token access_token refresh_token api_token auth_token
-      secret api_secret client_secret
-      api_key apikey private_key privatekey secret_key secretkey
-      credential auth authorization
-      encrypted encrypted_data
-      ssn social_security
-      credit_card card_number cvv cvc
-    ].freeze
+    # Sensitive-key pattern list. Delegates to SensitivePatterns (single
+    # source of truth shared with DatabaseCapturer / BulkDatabaseCapturer).
+    # Kept as a constant alias for backwards compatibility — code that
+    # used `Sanitizer::SENSITIVE_PATTERNS` continues to work.
+    SENSITIVE_PATTERNS = EzLogsAgent::SensitivePatterns::PATTERNS
 
     # Hard ceiling for nested object recursion. Deeper structures
     # collapse to the literal string "[Object]".
@@ -79,18 +72,13 @@ module EzLogsAgent
 
       # Check whether a key matches a sensitive pattern. Public so the
       # HTTP middleware can short-circuit early on identical keys.
+      # Delegates to SensitivePatterns (single source of truth — also
+      # consulted by DatabaseCapturer and BulkDatabaseCapturer).
       #
       # @param key [String, Symbol]
       # @return [Boolean]
       def sensitive_key?(key)
-        key_lower = key.to_s.downcase
-        return true if SENSITIVE_PATTERNS.any? { |pattern| key_lower.include?(pattern) }
-
-        user_patterns = EzLogsAgent.configuration.excluded_graphql_variable_keys || []
-        user_patterns.any? { |pattern| key_lower.include?(pattern.to_s.downcase) }
-      rescue
-        # Defensive: when in doubt, treat as sensitive.
-        true
+        EzLogsAgent::SensitivePatterns.match?(key)
       end
 
       private

data/lib/ez_logs_agent/sensitive_patterns.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+module EzLogsAgent
+  # Single source of truth for the agent's sensitive-key denylist. Used by
+  # every capture path that needs to mask a value based on its column /
+  # parameter / argument name (Sanitizer for HTTP params + job args,
+  # DatabaseCapturer for AR attributes, BulkDatabaseCapturer for SQL
+  # WHERE binds + SET values).
+  #
+  # This is a NAME-pattern denylist — the secondary defense, separate from
+  # the primary defense (Rails `encrypts :foo` introspection via
+  # `model.class.encrypted_attributes`, handled in EncryptedAttributes).
+  # Use both together: the encrypts check catches what the host app
+  # declared, this list catches what got past the declaration (legacy
+  # columns, manual hashing, externally-generated material).
+  #
+  # Matching rules:
+  # - Case-insensitive
+  # - Substring (so `customer_password` matches `password`)
+  # - User-extensible via `EzLogsAgent.configuration.excluded_graphql_variable_keys`
+  module SensitivePatterns
+    # Union of every column / key name we treat as sensitive. Curated
+    # from RFC 7468 / OWASP top sensitive-data categories plus
+    # ActiveRecord conventions. Keep this list narrow but defensive —
+    # adding a pattern is cheap; removing one is a backwards-incompatible
+    # behavior change for customer data on the wire.
+    PATTERNS = %w[
+      password passwd pwd
+      token access_token refresh_token api_token auth_token
+      secret api_secret client_secret
+      api_key apikey private_key privatekey secret_key secretkey
+      public_key signing_key
+      credential auth authorization
+      encrypted encrypted_data
+      pem cipher nonce salt digest signature hmac
+      ssn social_security
+      credit_card card_number cvv cvc
+    ].freeze
+
+    module_function
+
+    # @param key [String, Symbol, nil]
+    # @return [Boolean] true if the key matches a sensitive pattern OR
+    #   matches a user-configured pattern in `excluded_graphql_variable_keys`
+    def match?(key)
+      return false if key.nil?
+
+      key_lower = key.to_s.downcase
+      return true if PATTERNS.any? { |pattern| key_lower.include?(pattern) }
+
+      # Direct configuration access — any raise here propagates to the
+      # rescue below and we fail-closed. Wrapping the access in its own
+      # rescue would silently fall back to "no extra patterns" on a
+      # config bug and the outer rescue would never fire, which means
+      # the broken-config path becomes "leak", not "mask".
+      user_patterns = EzLogsAgent.configuration.excluded_graphql_variable_keys || []
+      user_patterns.any? { |pattern| key_lower.include?(pattern.to_s.downcase) }
+    rescue StandardError
+      # Defensive: if configuration access raises, treat as sensitive.
+      # Better to over-mask than to leak.
+      true
+    end
+  end
+end

data/lib/ez_logs_agent/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module EzLogsAgent
-  VERSION = "0.1.10"
+  VERSION = "0.2.0"
 end

data/lib/ez_logs_agent.rb

@@ -8,6 +8,9 @@ require_relative "ez_logs_agent/correlation"
 require_relative "ez_logs_agent/actor_validator"
 require_relative "ez_logs_agent/actor"
 require_relative "ez_logs_agent/user_agent_detector"
+require_relative "ez_logs_agent/sensitive_patterns"
+require_relative "ez_logs_agent/encrypted_attributes"
+require_relative "ez_logs_agent/bulk_sql_parser"
 require_relative "ez_logs_agent/sanitizer"
 require_relative "ez_logs_agent/event_builder"
 require_relative "ez_logs_agent/resource_extractor"
@@ -19,6 +22,7 @@ require_relative "ez_logs_agent/middleware/http_request"
 require_relative "ez_logs_agent/capturers/job_capturer"
 require_relative "ez_logs_agent/capturers/active_job_capturer"
 require_relative "ez_logs_agent/capturers/database_capturer"
+require_relative "ez_logs_agent/capturers/bulk_database_capturer"
 
 # Load Railtie only when Rails is present
 require_relative "ez_logs_agent/railtie" if defined?(Rails::Railtie)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ez_logs_agent
 version: !ruby/object:Gem::Version
-  version: 0.1.10
+  version: 0.2.0
 platform: ruby
 authors:
 - dezsirazvan
@@ -123,12 +123,15 @@ files:
 - lib/ez_logs_agent/actor.rb
 - lib/ez_logs_agent/actor_validator.rb
 - lib/ez_logs_agent/buffer.rb
+- lib/ez_logs_agent/bulk_sql_parser.rb
 - lib/ez_logs_agent/capturers/active_job_capturer.rb
+- lib/ez_logs_agent/capturers/bulk_database_capturer.rb
 - lib/ez_logs_agent/capturers/database_capturer.rb
 - lib/ez_logs_agent/capturers/job_capturer.rb
 - lib/ez_logs_agent/configuration.rb
 - lib/ez_logs_agent/configuration_validator.rb
 - lib/ez_logs_agent/correlation.rb
+- lib/ez_logs_agent/encrypted_attributes.rb
 - lib/ez_logs_agent/event_builder.rb
 - lib/ez_logs_agent/flush_scheduler.rb
 - lib/ez_logs_agent/logger.rb
@@ -137,6 +140,7 @@ files:
 - lib/ez_logs_agent/resource_extractor.rb
 - lib/ez_logs_agent/retry_sender.rb
 - lib/ez_logs_agent/sanitizer.rb
+- lib/ez_logs_agent/sensitive_patterns.rb
 - lib/ez_logs_agent/transport.rb
 - lib/ez_logs_agent/user_agent_detector.rb
 - lib/ez_logs_agent/version.rb