sixty 0.1.0

data/lib/sixty/instrument/mongo.rb

@@ -0,0 +1,482 @@
+# frozen_string_literal: true
+
+require_relative '../tracer'
+require_relative '../shape'
+require_relative '../stack'
+
+module Sixty
+  module Instrument
+    # MongoDB.
+    #
+    # The third data model this agent measures and the first that is not SQL.
+    # Every signal downstream — documents per call, calls per invocation,
+    # payload size, error rate — is defined on span attributes rather than on
+    # statements, so the feed, the detector and the MCP server need no change.
+    # What has to be rebuilt is everything below those attributes: identity,
+    # which has no statement to normalize, and the document count, which in a
+    # document database arrives in a different shape for every command.
+    #
+    # ── Why command monitoring here, and not in the JavaScript agent ──────────
+    #
+    # `@sixty-sh/node` deliberately refuses the driver's command events and
+    # patches `Collection.prototype` instead, because in Node a command event
+    # fires from the driver's own plumbing *after* the caller's async context is
+    # gone — every operation would be measured correctly and parented to
+    # nothing, and attribution is the whole product.
+    #
+    # The Ruby driver is synchronous. `started` and `succeeded` are published on
+    # the calling thread, inside the caller's own stack, so the current span is
+    # still the method that issued the query and the parenting is exact. That
+    # makes the official monitoring API strictly better here than monkey-patching
+    # would be: it is public, versioned, and covers Mongoid, the driver's own
+    # cursor round trips and commands issued through paths a collection patch
+    # would never see.
+    #
+    # ── One cursor is one operation, however many round trips it took ────────
+    #
+    # A `find` returning thirty thousand documents at the default batch size is
+    # not one exchange with the server: it is the initial command plus roughly
+    # three hundred `getMore` calls, each a network wait the application sits
+    # through. The naive reading of command monitoring records three hundred and
+    # one operations, and it is wrong in a way that misdirects the reader — the
+    # calling method appears to make three hundred database calls, which is the
+    # signature of an N+1 the code does not contain, and the fix a fanout finding
+    # suggests ("stop looping") is not the fix this needs ("set batchSize").
+    #
+    # So a cursor is held open: the `find` span stays alive until the cursor is
+    # exhausted, counting documents and round trips as its `getMore`s arrive, and
+    # is emitted once. `db_calls` then says one call, `rows` says thirty
+    # thousand, and `round_trips` — the signal the collector added for exactly
+    # this — says three hundred. That also matches what the JavaScript agent
+    # reports for the same query, which matters: the same regression in the same
+    # database should not look like two different findings depending on which
+    # language the service happens to be written in.
+    #
+    # ── No query plans ────────────────────────────────────────────────────────
+    #
+    # `explain` is the highest-value thing this adapter could add — COLLSCAN
+    # versus IXSCAN names a cause rather than a symptom. It is absent because
+    # Mongo can only explain a filter that still has its values in it, and this
+    # file drops values at the moment it sees them. Capturing a plan would mean
+    # *retaining* somebody's query values in order to compose a command out of
+    # them. That trade is refused here exactly as it is refused for MySQL.
+    module Mongo
+      # Commands the driver issues about itself. A handshake is not an operation
+      # anybody can act on, and heartbeats would otherwise be the highest-count
+      # "query" in every service.
+      IGNORED_COMMANDS = %w[
+        ismaster isMaster hello ping buildInfo getnonce authenticate saslStart
+        saslContinue logout endSessions getLog hostInfo listDatabases
+        connectionStatus getParameter
+      ].freeze
+
+      # Commands that can hand back a cursor rather than an answer.
+      CURSOR_COMMANDS = %w[find aggregate listIndexes listCollections].freeze
+
+      # Keys the driver adds to every command. They describe the session and the
+      # topology rather than the query, and including them would make the
+      # identity of every operation move with the driver's version.
+      #
+      # `cursor`, `batchSize` and `singleBatch` describe how the results are
+      # *fetched* rather than what was asked for, and they are excluded for a
+      # sharper reason than tidiness: setting or removing a batch size is
+      # precisely the change the `round_trips` signal exists to report, and an
+      # identity that moved with it would re-identify the operation at the
+      # moment of the change. The detector would then have nothing to compare —
+      # a new operation with no history beside an old one that stopped
+      # reporting — and the finding could never fire.
+      #
+      # `limit` is not in this list, and the difference is the point: a limit
+      # changes what you asked for.
+      ENVELOPE_KEYS = %w[
+        $db lsid txnNumber $clusterTime $readPreference $audit apiVersion
+        apiStrict apiDeprecationErrors signature startTransaction autocommit
+        readConcern writeConcern comment cursor batchSize singleBatch
+      ].freeze
+
+      # A command that never completes would otherwise leave its start context
+      # behind forever. The cap is per thread and far above any real number of
+      # commands in flight on one connection at one time.
+      MAX_IN_FLIGHT = 64
+      # And a cursor that is opened and never drained would leave a span open.
+      # Past this many, the oldest is emitted with what it has — an incomplete
+      # measurement rather than an unbounded one.
+      MAX_OPEN_CURSORS = 128
+      # The same protection in time rather than in count. The Ruby driver kills
+      # an abandoned cursor from a finalizer, so `break`ing out of a loop over a
+      # cursor releases nothing until the garbage collector gets to it — which
+      # may be never in a process that is not under memory pressure. A cursor
+      # nobody has read from in this long is reported with what it has.
+      MAX_CURSOR_AGE_SECONDS = 60
+
+      KEY = :sixty_mongo_in_flight
+
+      class << self
+        attr_reader :config, :subscriber
+
+        def install(config = nil)
+          return false if @installed
+          return false unless defined?(::Mongo::Monitoring::Global)
+
+          @config = config
+          @subscriber = Subscriber.new
+          # Global, so clients created later are covered. Clients that already
+          # exist — Mongoid builds its own during boot, which may be before
+          # this runs — are subscribed to individually below.
+          ::Mongo::Monitoring::Global.subscribe(::Mongo::Monitoring::COMMAND, @subscriber)
+          subscribe_existing_clients
+          Sixty.before_flush { sweep }
+          @installed = true
+        end
+
+        def installed?
+          @installed == true
+        end
+
+        # Exported for tests.
+        def reset!
+          @installed = false
+          @cursors = nil
+        end
+
+        # Cursors whose spans are still open, keyed by the server's cursor id.
+        #
+        # Process-wide rather than thread-local: a cursor may be handed to
+        # another thread to drain, and a span that could only be closed by the
+        # thread that opened it would leak in exactly that case.
+        def cursors
+          @cursors ||= {}
+        end
+
+        def cursor_mutex
+          @cursor_mutex ||= Mutex.new
+        end
+
+        # ---------------------------------------------------------------- state
+
+        def open_cursor(cursor_id, state, event)
+          span = Tracer.start_span(
+            kind: Tracer::KIND_DB,
+            name: state[:name],
+            attrs: { normalized_sql: state[:identity] }
+          )
+          span.attrs[:frames] = state[:frames] if state[:frames]
+          # Back-dated by the command's own duration, so the span covers the
+          # round trip that opened the cursor as well as the ones that follow.
+          span.start -= event.duration.to_f * 1000.0
+
+          entry = {
+            span: span,
+            rows: batch_length(event) || 0,
+            round_trips: 1,
+            bytes: bytes_from(event) || 0,
+            touched_at: Tracer.monotonic_ms
+          }
+          evicted = nil
+          cursor_mutex.synchronize do
+            cursors[cursor_id] = entry
+            evicted = cursors.shift if cursors.size > MAX_OPEN_CURSORS
+          end
+          close_entry(evicted[1], nil) if evicted
+          entry
+        end
+
+        def continue_cursor(cursor_id, event, error)
+          entry = cursor_mutex.synchronize { cursors[cursor_id] }
+          return false unless entry
+
+          entry[:round_trips] += 1
+          entry[:rows] += batch_length(event) || 0
+          entry[:bytes] += bytes_from(event) || 0
+          entry[:touched_at] = Tracer.monotonic_ms
+
+          if error || exhausted?(event)
+            cursor_mutex.synchronize { cursors.delete(cursor_id) }
+            close_entry(entry, error)
+          end
+          true
+        end
+
+        # Cursors nobody is reading any more, reported rather than held. Runs on
+        # the agent's flush thread, like every other piece of housekeeping here.
+        def sweep(max_age: MAX_CURSOR_AGE_SECONDS)
+          deadline = Tracer.monotonic_ms - (max_age * 1000.0)
+          stale = cursor_mutex.synchronize do
+            cursors.select { |_id, entry| entry[:touched_at] < deadline }
+                   .each_key { |id| cursors.delete(id) }
+          end
+          stale.each_value { |entry| close_entry(entry, nil) }
+          stale.size
+        end
+
+        # An abandoned cursor: the application stopped reading and the driver
+        # told the server so. The measurement ends where the reading did.
+        def kill_cursors(ids)
+          ids.each do |id|
+            entry = cursor_mutex.synchronize { cursors.delete(id) }
+            close_entry(entry, nil) if entry
+          end
+        end
+
+        def close_entry(entry, error)
+          span = entry[:span]
+          span.attrs[:rows] = entry[:rows]
+          span.attrs[:round_trips] = entry[:round_trips]
+          span.attrs[:bytes] = entry[:bytes] if entry[:bytes].positive?
+          Tracer.end_span(span, error)
+          Tracer.emit(span)
+        rescue StandardError
+          nil
+        end
+
+        # ------------------------------------------------------------ describing
+
+        # Everything about a command that has to be read before it runs.
+        def describe(event)
+          command = event.command || {}
+          command_name = event.command_name.to_s
+          collection = collection_for(command_name, command, event)
+          identity = identity_for(command_name, collection, command)
+
+          {
+            command_name: command_name,
+            name: "#{command_name}:#{collection}",
+            identity: identity,
+            frames: Stack.capture(identity),
+            cursor_id: command_name == 'getMore' ? cursor_id_of(command['getMore']) : nil
+          }
+        end
+
+        # The identity of the operation: the command, what it ran against, and
+        # the *structure* of its arguments. See Sixty::Shape — no value in the
+        # command has a path into this string.
+        def identity_for(command_name, collection, command)
+          parts = []
+          # A pipeline is ordered structure, so every stage counts and in order.
+          pipeline = command['pipeline'] || command[:pipeline]
+          parts << Shape.shape_of_sequence(pipeline) if pipeline.is_a?(Array)
+
+          rest = command.reject do |key, _value|
+            name = key.to_s
+            ENVELOPE_KEYS.include?(name) || name == command_name || name == 'pipeline'
+          end
+          shape = Shape.shape_of(rest)
+          parts << "{#{shape}}" unless shape.empty?
+
+          "#{command_name} #{collection} #{parts.join(' ')}".strip
+        end
+
+        # What the command ran against. For most commands the collection is the
+        # value of the command key itself (`{find: 'orders', ...}`); `getMore`
+        # names a cursor there and carries the collection separately.
+        def collection_for(command_name, command, event)
+          value = command_name == 'getMore' ? command['collection'] : command[command_name]
+          return value if value.is_a?(String) && !value.empty?
+
+          database = event.database_name.to_s
+          database.empty? ? 'collection' : database
+        end
+
+        # -------------------------------------------------------------- replies
+
+        # Documents returned for a read, documents affected for a write — the
+        # same meaning `rows` has for every SQL client in this directory. A
+        # `rows` drift that meant one thing on Postgres and another on Mongo
+        # would be a chart nobody can read.
+        def rows_from(event, command_name)
+          batch = batch_length(event)
+          return batch if batch
+
+          reply = reply_of(event)
+          return nil unless reply
+
+          case command_name
+          when 'count', 'countDocuments'
+            # A count returns one number, so one document came back. Reporting
+            # the count itself would say that counting a million-document
+            # collection returned a million documents, and the first collection
+            # to grow would look like the regression this product exists to
+            # report.
+            1
+          when 'distinct'
+            values = reply['values']
+            values.is_a?(Array) ? values.length : 1
+          when 'findAndModify'
+            reply['value'].nil? ? 0 : 1
+          when 'update'
+            number(reply['nModified']) || number(reply['n'])
+          else
+            number(reply['n'])
+          end
+        end
+
+        def batch_length(event)
+          cursor = reply_of(event)&.[]('cursor')
+          return nil unless cursor.is_a?(Hash)
+
+          batch = cursor['firstBatch'] || cursor['nextBatch']
+          batch.is_a?(Array) ? batch.length : nil
+        end
+
+        def cursor_id_from_reply(event)
+          cursor = reply_of(event)&.[]('cursor')
+          cursor.is_a?(Hash) ? cursor_id_of(cursor['id']) : nil
+        end
+
+        def exhausted?(event)
+          id = cursor_id_from_reply(event)
+          id.nil? || id.zero?
+        end
+
+        # BSON::Int64 and Integer both appear here depending on driver version
+        # and platform, and they are not `eql?` — so a map keyed by the raw value
+        # would miss on lookup and every cursor would look like a new one.
+        def cursor_id_of(value)
+          return nil if value.nil?
+
+          value.respond_to?(:value) ? value.value.to_i : value.to_i
+        rescue StandardError
+          nil
+        end
+
+        # A failed command has no reply at all — `CommandFailed` carries a
+        # message where `CommandSucceeded` carries the server's answer. Reading
+        # it unguarded is how the first version of this file dropped every failed
+        # query: the error reached the application correctly and the operation
+        # that produced it was never recorded.
+        def reply_of(event)
+          return nil unless event.respond_to?(:reply)
+
+          reply = event.reply
+          reply.is_a?(Hash) ? reply : nil
+        end
+
+        # Sampled, because serializing a whole reply to measure it would cost
+        # more than the query. Precision does not matter: this exists to catch a
+        # payload going from 4KB to 2MB.
+        def bytes_from(event)
+          cursor = reply_of(event)&.[]('cursor')
+          batch = cursor.is_a?(Hash) ? (cursor['firstBatch'] || cursor['nextBatch']) : nil
+          return nil unless batch.is_a?(Array) && !batch.empty?
+
+          # Three documents rather than five: serializing a BSON document to
+          # measure it is the most expensive thing this file does per query, and
+          # the number only has to be right to an order of magnitude.
+          sample = batch.first(3)
+          sampled = sample.sum { |doc| doc.to_bson.length }
+          ((sampled.to_f / sample.length) * batch.length).round
+        rescue StandardError
+          nil
+        end
+
+        def number(value)
+          value.is_a?(Numeric) ? value.to_i : nil
+        end
+
+        private
+
+        # Mongoid registers its clients before an initializer runs, and a global
+        # subscription only reaches clients constructed afterwards. Reaching into
+        # `Mongoid.clients` is guarded twice over: the constant may not exist,
+        # and a driver that changed the shape of it must not take the boot down.
+        def subscribe_existing_clients
+          return unless defined?(::Mongoid) && ::Mongoid.respond_to?(:clients)
+
+          ::Mongoid.clients.each_key do |name|
+            ::Mongoid.client(name).subscribe(::Mongo::Monitoring::COMMAND, @subscriber)
+          rescue StandardError
+            next
+          end
+        rescue StandardError
+          nil
+        end
+      end
+
+      # The monitoring contract: three methods, none of which may raise into the
+      # driver. A subscriber that throws would fail the application's query.
+      class Subscriber
+        def started(event)
+          return unless Sixty.enabled?
+
+          command_name = event.command_name.to_s
+          # Not an operation, but the end of one: the application stopped
+          # reading a cursor and the driver is telling the server so.
+          if command_name == 'killCursors'
+            Mongo.kill_cursors(Array(event.command['cursors']).map { |id| Mongo.cursor_id_of(id) }.compact)
+            return
+          end
+          return if IGNORED_COMMANDS.include?(command_name)
+
+          in_flight = (Thread.current[KEY] ||= {})
+          in_flight.clear if in_flight.size >= MAX_IN_FLIGHT
+          in_flight[event.request_id] = Mongo.describe(event)
+        rescue StandardError
+          nil
+        end
+
+        def succeeded(event)
+          finish(event, nil)
+        end
+
+        def failed(event)
+          # `event.message` is the server's error text. It is used as the error
+          # message and never as part of the operation's identity, so a failure
+          # that quotes a value cannot mint an operation from it.
+          finish(event, CommandFailed.new(event.message.to_s))
+        end
+
+        private
+
+        def finish(event, error)
+          in_flight = Thread.current[KEY]
+          state = in_flight&.delete(event.request_id)
+          return unless state
+
+          # A `getMore` belongs to the cursor that opened it, not to itself.
+          # Only a cursor this agent never saw opened — one already in flight
+          # when the process started measuring — is recorded on its own.
+          if state[:command_name] == 'getMore'
+            return if Mongo.continue_cursor(state[:cursor_id], event, error)
+          elsif error.nil? && CURSOR_COMMANDS.include?(state[:command_name])
+            cursor_id = Mongo.cursor_id_from_reply(event)
+            if cursor_id && !cursor_id.zero?
+              Mongo.open_cursor(cursor_id, state, event)
+              return
+            end
+          end
+
+          record(state, event, error)
+        rescue StandardError
+          nil
+        end
+
+        # A command whose answer is complete when it returns: a write, a count,
+        # or a read that fitted in one batch.
+        def record(state, event, error)
+          attrs = { normalized_sql: state[:identity], round_trips: 1 }
+          attrs[:frames] = state[:frames] if state[:frames]
+          rows = Mongo.rows_from(event, state[:command_name])
+          attrs[:rows] = rows if rows
+          bytes = Mongo.bytes_from(event)
+          attrs[:bytes] = bytes if bytes
+
+          Tracer.record(
+            kind: Tracer::KIND_DB,
+            name: state[:name],
+            # The driver reports seconds; everything in this agent is
+            # milliseconds, and a unit disagreement here would put Mongo
+            # operations three orders of magnitude below every other query in
+            # the same feed.
+            duration_ms: event.duration.to_f * 1000.0,
+            attrs: attrs,
+            error: error
+          )
+        end
+      end
+
+      # The error type recorded for a command the server refused. Named rather
+      # than a bare StandardError so the feed says what kind of failure it was.
+      class CommandFailed < StandardError; end
+    end
+  end
+end

data/lib/sixty/instrument/mysql.rb

@@ -0,0 +1,187 @@
+# frozen_string_literal: true
+
+require_relative '../tracer'
+require_relative '../sql'
+require_relative '../stack'
+
+module Sixty
+  module Instrument
+    # The MySQL drivers — `mysql2` and `trilogy` — patched directly.
+    #
+    # Same audience as the `pg` patch: everything that is not ActiveRecord.
+    # A Rails app on MySQL is covered by `sql.active_record`, which is public
+    # API and does not care which driver is underneath; this covers Sinatra,
+    # Sequel, workers and scripts, both of whose drivers reach the server
+    # through one method each.
+    #
+    # ── The dialect is not a detail ──────────────────────────────────────────
+    #
+    # Everything here normalizes with `:mysql`, because `"alice@example.com"` is
+    # a *string literal* in MySQL's default sql_mode and a quoted *identifier*
+    # in Postgres. Reading a MySQL statement with the Postgres rules would keep
+    # it — the one class of bug this agent must never have.
+    #
+    # ── Rows, and why the result is not iterated to find them ────────────────
+    #
+    # `Mysql2::Result` is lazily populated when the client is streaming, and
+    # asking a streaming result for its size consumes it. The count is therefore
+    # only read where it is free — a buffered result already knows it — and a
+    # streaming query reports no row count rather than a wrong one, or a query
+    # this agent silently drained on the application's behalf.
+    module Mysql
+      class << self
+        def install(config = nil)
+          @config = config
+          installed = false
+          installed |= install_mysql2
+          installed |= install_trilogy
+          installed
+        end
+
+        def installed?
+          @mysql2_installed == true || @trilogy_installed == true
+        end
+
+        def reset!
+          @mysql2_installed = false
+          @trilogy_installed = false
+        end
+
+        # One span for one statement, shared by both drivers.
+        #
+        # `count_rows` is decided by the caller rather than here, because only
+        # the patch can see the query options that say whether the result will
+        # be streamed — and asking a streaming result for its size would consume
+        # it, which is a change in the application's behaviour rather than a
+        # measurement of it.
+        def measure(sql, connection: nil, count_rows: true)
+          return yield unless Sixty.enabled? && sql.is_a?(String) && !sql.empty?
+
+          normalized = Sql.normalize_sql(sql, :mysql)
+          started = Tracer.monotonic_ms
+          begin
+            result = yield
+          rescue StandardError => e
+            record(normalized, Tracer.monotonic_ms - started, nil, nil, false, e)
+            raise
+          end
+          record(normalized, Tracer.monotonic_ms - started, result, connection, count_rows, nil)
+          result
+        end
+
+        private
+
+        def install_mysql2
+          return false if @mysql2_installed
+          return false unless defined?(::Mysql2::Client)
+
+          ::Mysql2::Client.prepend(Mysql2Patch)
+          ::Mysql2::Client.prepend(Mysql2PreparePatch)
+          ::Mysql2::Statement.prepend(Mysql2StatementPatch) if defined?(::Mysql2::Statement)
+          @mysql2_installed = true
+        end
+
+        def install_trilogy
+          return false if @trilogy_installed
+          return false unless defined?(::Trilogy)
+
+          ::Trilogy.prepend(TrilogyPatch)
+          @trilogy_installed = true
+        end
+
+        def record(normalized, duration_ms, result, connection, count_rows, error)
+          attrs = { normalized_sql: normalized }
+          frames = Stack.capture(normalized)
+          if frames
+            attrs[:frames] = frames
+            attrs[:file] = frames.first[:file]
+            attrs[:line] = frames.first[:line]
+          end
+          rows = row_count(result, connection, count_rows)
+          attrs[:rows] = rows if rows
+          attrs[:fields] = result.fields.length if result.respond_to?(:fields) && result.fields
+
+          Tracer.record(
+            kind: Tracer::KIND_DB,
+            name: Sql.sql_operation_name(normalized),
+            duration_ms: duration_ms,
+            attrs: attrs,
+            error: error
+          )
+          # No plans on MySQL. See Sixty::Plans#enqueue: there is no
+          # GENERIC_PLAN here, so an EXPLAIN would need the values bound.
+        rescue StandardError
+          # An unmeasured query, not a failed one.
+          nil
+        end
+
+        def row_count(result, connection, count_rows)
+          # A read: the rows the statement returned.
+          return result.count if count_rows && result.respond_to?(:count)
+
+          # A write returns no result set and reports what it changed instead.
+          # Both are "rows this statement was responsible for", which is what
+          # this signal means for every other client in this directory.
+          return result.affected_rows if result.respond_to?(:affected_rows)
+          return connection.affected_rows if connection.respond_to?(:affected_rows)
+
+          nil
+        rescue StandardError
+          nil
+        end
+      end
+
+      # A prepared statement runs its SQL somewhere other than the call site:
+      # `client.prepare(sql)` has the text and `statement.execute(id)` has the
+      # values, and only the second one is a query. Without this, an application
+      # using prepared statements reports no database calls at all — the silent
+      # blind spot this project refuses, since nothing errors and the feed
+      # simply shows a service that never touches its database.
+      #
+      # The text is carried on the statement object itself rather than in a map:
+      # a statement is a Ruby object with the same lifetime as the prepared
+      # statement it stands for, so there is nothing to bound and nothing to
+      # evict.
+      module Mysql2PreparePatch
+        def prepare(*args, &block)
+          statement = super
+          statement.instance_variable_set(:@sixty_sql, args.first)
+          statement
+        rescue StandardError
+          # A driver that changed the shape of `prepare` must not take the
+          # application's query with it.
+          statement
+        end
+        ruby2_keywords :prepare
+      end
+
+      module Mysql2StatementPatch
+        def execute(*args, &block)
+          sql = instance_variable_get(:@sixty_sql)
+          Sixty::Instrument::Mysql.measure(sql) { super }
+        end
+        ruby2_keywords :execute
+      end
+
+      module Mysql2Patch
+        def query(*args, &block)
+          # `stream: true` can come from the call or from the client's defaults,
+          # and either one means the rows are not there yet.
+          options = args[1].is_a?(Hash) ? args[1] : {}
+          streaming = options[:stream] || (query_options.is_a?(Hash) && query_options[:stream])
+          Sixty::Instrument::Mysql.measure(
+            args.first, connection: self, count_rows: !streaming
+          ) { super }
+        end
+        ruby2_keywords :query
+      end
+
+      module TrilogyPatch
+        def query(*args, &block)
+          Sixty::Instrument::Mysql.measure(args.first, connection: self) { super }
+        end
+        ruby2_keywords :query
+      end
+    end
+  end
+end