rabbitmq-graph 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 0c207d8f0935bddfd81f0e6600182134797df3a27f283a4b71ac172f8ea23549
+  data.tar.gz: 75690fe90096674172429863f3bb21b71a9b11cfa1014fbd1f84acf4d85fded0
+SHA512:
+  metadata.gz: f381610c65e306db2d734852f21d3cd23932747418cea7437d8ceffbc94a02feec3f8215ff4f95910e0b7209c1ff2bb4b2119ca8c9a8ee0d6c87707e7478668f
+  data.tar.gz: 9a2013484acbac11866864c1095b8a795210ac8e1d6c77478fd869ae5e535dd7a40a1ebdcc057c7208aa71e9ae4e3e800a8aa3a3ec06f168c542edfd1bf8cb48

data/README.md

@@ -0,0 +1,81 @@
+# rabbitmq-graph
+
+[![Dependency Status](https://gemnasium.com/badges/github.com/sldblog/rabbitmq-graph.svg)](https://gemnasium.com/github.com/sldblog/rabbitmq-graph)
+[![CircleCI](https://circleci.com/gh/sldblog/rabbitmq-graph.svg?style=svg&circle-token=68531f42debaa4ff5b3bddb62a4672ca2eaabaf4)](https://circleci.com/gh/sldblog/rabbitmq-graph)
+[![Maintainability](https://api.codeclimate.com/v1/badges/146dab10c24b4dd7b75e/maintainability)](https://codeclimate.com/github/sldblog/rabbitmq-graph/maintainability)
+[![Test Coverage](https://api.codeclimate.com/v1/badges/146dab10c24b4dd7b75e/test_coverage)](https://codeclimate.com/github/sldblog/rabbitmq-graph/test_coverage)
+
+Discover RabbitMQ topology.
+
+## Assumptions
+
+- Routing keys are segmented with dots (`.`).
+
+  | Segment name | Routing key | Extracted | Assumed to be |
+  | --- | --- | --- | --- |
+  | from\_app | **splitter**.experiment.something.assigned | splitted | The name of the publishing application. |
+  | entity | splitter.**experiment**.something.assigned | experiment | The entity that is participating in the action. |
+  | actions | splitter.experiment.**something.assigned** | something.assigned | The action(s) describing the event. |
+
+- [Consumer tags][hutch-consumer-tag-pr] are configured to contain the name of the consuming application.
+
+## How to run?
+
+Without arguments `bin/rabbitmq-graph` will connect to `localhost:15672` with the default guest user.
+
+### Configuration
+
+| Setting | Configuration | Effect | Default |
+| ------- | ------------- | ------ | ------- |
+| RabbitMQ management URL | `-uURL`<br/>`--url=URL`<br/>or environment variable<br/>`RABBITMQ_API_URI` | Specifies the connection URL to RabbitMQ management API | http://guest:guest@localhost:15672/ |
+| Save topology | `--save-topology=FILE` | After discovery save the topology to the given file. | disabled |
+| Read topology | `--read-topology=FILE` | Skip discovery and use a stored topology file. | disabled |
+| Choose format | `--format=FORMAT` | Choose an output format. `--help` will give a list of available options. | `DotFormat` |
+
+#### Dot format specific options
+
+| Setting | Configuration | Effect | Default |
+| ------- | ------------- | ------ | ------- |
+| Show only applications | `--dot-applications-only` | Creates a graph without entity nodes. | disabled |
+| Label details | `--dot-label-detail=DETAILS` | Comma separated segment names to display on labels drawn between applications and/or entities. | `'actions'` |
+
+### Show only applications
+
+- **enabled**: will only show application to application relations.
+- **disabled** (default): will show application to entity to application relations. The edge going into the entity and coming out of the entity will have the same label.
+
+### Label details
+
+Affects the labeling of edges:
+
+- `'entity,actions'`: displays the entity name and the actions on the edge.
+- `'entity'`: displays the entity name on the edge.
+- `'actions'`: displays the actions on the edge.
+- `''` (empty string): displays no labels.
+
+Any combination and order of the above is allowed.
+
+### Example
+
+Running the discovery against a dockerised `rabbitmq:3.6-management`:
+
+```
+$ docker run --detach --publish 5672:5672 --publish 15672:15672 rabbitmq:3.6-management
+
+$ RABBITMQ_API_URI=http://localhost:15672/ bin/rabbitmq-graph > test.dot
+I, [2018-04-30T13:05:29.735060 #90042]  INFO -- : connecting to rabbitmq HTTP API (http://guest@127.0.0.1:15672/)
+Discovering bindings: |================================================================================================|
+Discovering queues: |==================================================================================================|
+
+$ fdp -O -Tpng test.dot   # assumes "graphviz" is installed
+$ open test.dot.png
+```
+
+## How to configure consumer tags?
+
+### hutch
+
+Hutch supports consumer tag prefixes since [0.24][hutch-0.24].
+
+[hutch-consumer-tag-pr]: https://github.com/gocardless/hutch/pull/265
+[hutch-0.24]: https://github.com/gocardless/hutch/blob/master/CHANGELOG.md#0240--february-1st-2017

data/bin/rabbitmq-graph

@@ -0,0 +1,75 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'json'
+require 'optparse'
+require 'rabbitmq-graph/discover'
+require 'rabbitmq-graph/dot_format'
+require 'rabbitmq-graph/markdown_table_format'
+require 'rabbitmq-graph/route'
+
+def topology(options)
+  if (read_file = options[:discover][:read_topology_file])
+    JSON.parse(IO.read(read_file), symbolize_names: true).map { |route_hash| Route.new(route_hash) }
+  else
+    result = Discover.new(api_url: options[:discover][:api_url]).topology
+    if (save_file = options[:discover][:save_topology_file])
+      output = "[\n  " + result.map { |e| JSON.generate(e.to_h) }.join(",\n  ") + "\n]\n"
+      IO.write(save_file, output)
+    end
+    result
+  end
+end
+
+def setup_discovery_options(options, option_parser)
+  options[:discover] ||= {}
+  options[:discover][:api_url] = ENV['RABBITMQ_API_URI'] || 'http://guest:guest@localhost:15672/'
+  option_parser.on('-uURL', '--url=URL', 'RabbitMQ management API URL. ' \
+                   'Defaults to "http://guest:guest@localhost:15672/". ' \
+                   'Also configurable through the "RABBITMQ_API_URI" environment variable.') do |url|
+    options[:discover][:api_url] = url
+  end
+  option_parser.on('--read-topology=FILE', 'Skip discovery and use a stored topology file.') do |file|
+    options[:discover][:read_topology_file] = file
+  end
+  option_parser.on('--save-topology=FILE', 'After discovery save the topology to the given file.') do |file|
+    options[:discover][:save_topology_file] = file
+  end
+end
+
+def setup_format_options(options, option_parser, formats:)
+  options[:format] = formats.first
+  option_parser.on('--format=FORMAT', formats.map(&:to_s), "Select format to use from #{formats.join(', ')}. " \
+                   "Defaults to #{options[:format]}.") do |format|
+    options[:format] = Object.const_get(format)
+  end
+
+  formats.each { |format| options[format] = {} }
+
+  # DotFormat specific options
+  options[DotFormat][:show_entities] = true
+  option_parser.on('--dot-applications-only', 'Creates a graph without entity nodes.') do |apps_only|
+    options[DotFormat][:show_entities] = !apps_only
+  end
+
+  options[DotFormat][:label_detail] = %i[actions]
+  option_parser.on('--dot-label-detail=DETAILS', 'Specifies edge label format. ' \
+                   'Comma separated list of "queue_name", "entity", "actions"') do |label_detail|
+    options[DotFormat][:label_detail] = label_detail.to_s.split(',').map(&:strip).reject(&:empty?).map(&:to_sym)
+  end
+end
+
+options = {}
+OptionParser.new do |option_parser|
+  option_parser.banner = "Usage: #{File.basename(__FILE__)} [options]"
+  setup_discovery_options(options, option_parser)
+  setup_format_options(options, option_parser, formats: [DotFormat, MarkdownTableFormat])
+  option_parser.on('-h', '--help', 'Prints this help.') do
+    puts option_parser
+    exit
+  end
+end.parse!
+
+chosen_format = options[:format]
+format_options = options[chosen_format].merge(topology: topology(options))
+puts chosen_format.new(format_options).present

data/lib/rabbitmq-graph.rb

@@ -0,0 +1 @@
+# frozen_string_literal: true

data/lib/rabbitmq-graph/discover.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+require 'hutch'
+require 'json'
+require 'logger'
+require 'ruby-progressbar'
+require 'uri'
+require 'rabbitmq-graph/route'
+
+# Using RabbitMQ's HTTP management API, discovers the server's publisher/subscriber topology.
+#
+# Assumes that:
+#  - `consumer_tag`s are set on consumers to the consuming application's name
+#  - bound routing keys are in the format of `application_name.entity_name[.action]+`
+class Discover
+  def initialize(api_url: ENV.fetch('RABBITMQ_API_URI', 'http://guest:guest@localhost:15672/'),
+                 api_client: nil, output: $stderr)
+    @output_io = output
+    Hutch::Logging.logger = Logger.new(output_io)
+    configure_hutch_http_api(api_url)
+    configure_api_client(api_client)
+  end
+
+  def topology
+    all_publishers = discover_routing_keys
+    all_consumers = discover_queues_and_consumers
+    queue_names = all_publishers.keys | all_consumers.keys
+
+    queue_names.inject([]) do |result, queue_name|
+      publishers = all_publishers[queue_name] || []
+      publishers << {} if publishers.empty?
+      consumers = all_consumers[queue_name] || []
+      consumers << {} if consumers.empty?
+
+      routes = publishers
+               .flat_map { |route| consumers.map { |consumer| route.merge(consumer) } }
+               .map { |route| route.delete_if { |_key, value| value.nil? } }
+               .map { |route| Route.new(route.merge(queue_name: queue_name)) }
+               .uniq
+      result.concat(routes)
+    end
+  end
+
+  private
+
+  attr_reader :client, :output_io
+
+  def configure_hutch_http_api(api_url)
+    parsed_uri = URI(api_url)
+    Hutch::Config.set(:mq_api_host, parsed_uri.host)
+    Hutch::Config.set(:mq_username, parsed_uri.user || 'guest')
+    Hutch::Config.set(:mq_password, parsed_uri.password || 'guest')
+    Hutch::Config.set(:mq_api_port, parsed_uri.port)
+    Hutch::Config.set(:mq_api_ssl, parsed_uri.scheme == 'https')
+  end
+
+  def configure_api_client(api_client)
+    return @client = api_client if api_client
+
+    broker = Hutch::Broker.new
+    broker.set_up_api_connection
+    @client = broker.api_client
+  end
+
+  def discover_routing_keys
+    items = {}
+    ProgressBar.create(title: 'Discovering bindings', total: client.bindings.size, output: output_io).tap do |progress|
+      bindings.each do |mq_binding|
+        queue_name = mq_binding[:queue_name]
+        items[queue_name] ||= []
+        items[queue_name] << { routing_key: mq_binding[:routing_key] }
+        progress.increment
+      end
+      progress.finish
+    end
+    items
+  end
+
+  def discover_queues_and_consumers
+    items = {}
+    ProgressBar.create(title: 'Discovering queues', total: client.queues.size, output: output_io).tap do |progress|
+      bound_queues.each do |queue|
+        bound_consumers(queue).each do |consumer|
+          queue_name = consumer[:queue_name]
+          items[queue_name] ||= []
+          items[queue_name] << { consumer_tag: consumer[:consumer] }
+        end
+        progress.increment
+      end
+      progress.finish
+    end
+    items
+  end
+
+  def bindings
+    client.bindings.lazy
+          .select { |binding| binding['destination_type'] == 'queue' }
+          .reject { |binding| binding['routing_key'].empty? }
+          .reject { |binding| binding['source'].empty? }
+          .map    { |binding_data| extract_binding_data(binding_data) }
+  end
+
+  def bound_queues
+    client.queues.lazy
+          .map { |queue| fetch_queue_data(queue['vhost'], queue['name']) }
+          .map { |queue| queue['consumer_details'] }
+  end
+
+  def bound_consumers(queue_data)
+    queue_data
+      .flatten
+      .reject(&:empty?)
+      .map { |consumer_data| extract_consumer_data(consumer_data) }
+  end
+
+  def extract_binding_data(binding_data)
+    {
+      vhost: binding_data['vhost'],
+      queue_name: binding_data['destination'],
+      routing_key: binding_data['routing_key']
+    }
+  end
+
+  def extract_consumer_data(consumer_data)
+    {
+      vhost: consumer_data['queue']['vhost'],
+      queue_name: consumer_data['queue']['name'],
+      consumer: consumer_data['consumer_tag']
+    }
+  end
+
+  def fetch_queue_data(vhost, name)
+    escaped_vhost_path = URI.encode_www_form_component(vhost)
+    JSON.parse(client.query_api(path: "/queues/#{escaped_vhost_path}/#{name}").body)
+  end
+end

data/lib/rabbitmq-graph/dot_format.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+require 'set'
+
+# Presents a RabbitMQ topology in graphviz's .dot format
+class DotFormat
+  def initialize(topology:, show_entities: true, label_detail: %i[actions])
+    @topology = topology
+    @show_entities = show_entities
+    @label_detail = label_detail
+  end
+
+  def present
+    <<-GRAPH
+    digraph G {
+      #{render_application_subgraph}
+      #{render_entity_subgraph}
+      #{message_edges.join("\n")}
+    }
+    GRAPH
+  end
+
+  private
+
+  attr_reader :topology, :show_entities, :label_detail
+
+  def render_application_subgraph
+    <<-APPS
+      subgraph Apps {
+        node [shape=oval fillcolor=yellow style=filled]
+        #{application_nodes.join("\n")}
+      }
+    APPS
+  end
+
+  def render_entity_subgraph
+    return '' unless show_entities
+    <<-ENTITIES
+      subgraph Entities {
+        node [shape=box fillcolor=turquoise style=filled]
+        #{entity_nodes.join("\n")}
+      }
+    ENTITIES
+  end
+
+  def application_nodes
+    applications = {}
+    topology.each do |route|
+      applications[route.source_app] ||= Set.new
+      applications[route.source_app] << 'fillcolor="red"' if route.missing_source?
+
+      applications[route.target_app] ||= Set.new
+      applications[route.target_app] << 'fillcolor="red"' if route.missing_target?
+      applications[route.target_app] << 'fillcolor="orange"' if route.default_consumer_tag?
+    end
+    applications.map { |name, properties| %("#{name}" [#{properties.to_a.join(' ')}]) }.sort
+  end
+
+  def entity_nodes
+    entities = topology.map(&:entity).sort.uniq
+    entities.map { |entity| %("#{entity}") }
+  end
+
+  def message_edges
+    topology.map { |route| %(#{route_path(route)} [#{route_properties(route)}]) }.uniq
+  end
+
+  def route_path(route)
+    path = []
+    path << route.source_app
+    path << route.entity if show_entities
+    path << route.target_app
+    path.map { |text| %("#{text}") }.join('->')
+  end
+
+  def route_properties(route)
+    label = label_detail.select { |detail| route.respond_to?(detail) }
+                        .map { |detail| route.public_send(detail) }
+                        .flatten.join('.')
+    properties = []
+    properties << %(label="#{label}")
+    properties << %(color="red") if route.missing_source? || route.missing_target?
+    properties.join(' ')
+  end
+end

data/lib/rabbitmq-graph/markdown_table_format.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+# Presents a RabbitMQ topology in a GitHub-flavoured Markdown table
+class MarkdownTableFormat
+  def initialize(topology:)
+    @topology = topology
+  end
+
+  def present
+    no_consumer_routes = topology.select(&:missing_target?)
+    no_binding_routes = topology.select(&:missing_source?)
+    default_tag_routes = topology.select(&:default_consumer_tag?)
+    connected_routes = topology.reject { |r| r.missing_source? || r.missing_target? || r.default_consumer_tag? }
+
+    lines = []
+    lines.concat(route_table('Routes without consumers', no_consumer_routes))
+    lines.concat(route_table('Routes without publisher bindings', no_binding_routes))
+    lines.concat(route_table('Routes with default consumer names', default_tag_routes))
+    lines.concat(route_table('Named, connected routes', connected_routes))
+    lines.join("\n")
+  end
+
+  private
+
+  attr_reader :topology
+
+  def route_table(title, routes)
+    return [] if routes.empty?
+    lines = []
+    lines << ''
+    lines << "# #{title}"
+    lines << ''
+    lines << '| Publisher application | Consumer application | Entity | Actions | Queue |'
+    lines << '| --- | --- | --- | --- | --- |'
+    lines.concat(routes.map { |route| route_line(route) }.uniq)
+  end
+
+  def route_line(route)
+    columns = []
+    columns << route.source_app
+    columns << route.target_app
+    columns << route.entity
+    columns << route.actions.join('.')
+    columns << route.queue_name
+    '| ' + columns.join(' | ') + ' |'
+  end
+end

data/lib/rabbitmq-graph/route.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+# Extracts publisher/consumer application names and routing key fragments from routing data
+class Route
+  DEFAULT_CONSUMER_TAG = 'default-consumer-tag'
+  MISSING_SOURCE_LABEL = 'no-routing-key-binding'
+  MISSING_TARGET_LABEL = 'no-consumers'
+
+  def initialize(queue_name:, routing_key: nil, consumer_tag: nil)
+    @queue_name = queue_name
+    @routing_fragments = routing_key.to_s.split('.')
+    @consumer_tag = consumer_tag
+
+    @source_app = routing_fragments[0] || MISSING_SOURCE_LABEL
+    @entity = routing_fragments[1] || ''
+    @actions = routing_fragments[2..-1] || []
+    @target_app = consumer_tag_to_application_name(consumer_tag) || MISSING_TARGET_LABEL
+  end
+
+  attr_reader :source_app, :target_app, :entity, :actions, :queue_name
+
+  def missing_source?
+    source_app == MISSING_SOURCE_LABEL
+  end
+
+  def missing_target?
+    target_app == MISSING_TARGET_LABEL
+  end
+
+  def default_consumer_tag?
+    target_app == DEFAULT_CONSUMER_TAG
+  end
+
+  def to_h
+    { queue_name: queue_name, routing_key: routing_fragments.join('.'), consumer_tag: consumer_tag }
+  end
+
+  def ==(other)
+    eql?(other)
+  end
+
+  def eql?(other)
+    [queue_name, routing_fragments, consumer_tag] == [other.queue_name, other.routing_fragments, other.consumer_tag]
+  end
+
+  protected
+
+  attr_reader :routing_fragments, :consumer_tag
+
+  private
+
+  def consumer_tag_to_application_name(tag)
+    return DEFAULT_CONSUMER_TAG if tag =~ /^bunny-/ || tag =~ /^hutch-/ || tag =~ /^amq\.ctag/
+    return tag.split('-')[0..-6].join('-') if tag =~ /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i
+    return tag.split('-')[0..-3].join('-') if tag =~ /[0-9]+-[0-9]+$/
+    tag
+  end
+end

data/rabbitmq-graph.gemspec

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+Gem::Specification.new do |s|
+  s.name        = 'rabbitmq-graph'
+  s.version     = '0.1.1'
+  s.summary     = 'Discover RabbitMQ topology'
+  s.description = 'Map out RabbitMQ topology with the use of routing key conventions and consumer tags.'
+  s.authors     = ['David Lantos']
+  s.email       = 'david.lantos+rabbitmq-graph@gmail.com'
+  s.bindir      = 'bin'
+  s.executables = ['rabbitmq-graph']
+  s.files       = `git ls-files lib bin *.gemspec *.md`.split
+  s.homepage    = 'https://github.com/sldblog/rabbitmq-graph'
+  s.metadata    = { 'source_code_uri' => 'https://github.com/sldblog/rabbitmq-graph' }
+
+  s.add_runtime_dependency 'hutch', '~> 0.24'
+  s.add_runtime_dependency 'ruby-progressbar', '~> 1.9'
+end

metadata

@@ -0,0 +1,81 @@
+--- !ruby/object:Gem::Specification
+name: rabbitmq-graph
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- David Lantos
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2018-05-09 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: hutch
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.24'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.24'
+- !ruby/object:Gem::Dependency
+  name: ruby-progressbar
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.9'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.9'
+description: Map out RabbitMQ topology with the use of routing key conventions and
+  consumer tags.
+email: david.lantos+rabbitmq-graph@gmail.com
+executables:
+- rabbitmq-graph
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- bin/rabbitmq-graph
+- lib/rabbitmq-graph.rb
+- lib/rabbitmq-graph/discover.rb
+- lib/rabbitmq-graph/dot_format.rb
+- lib/rabbitmq-graph/markdown_table_format.rb
+- lib/rabbitmq-graph/route.rb
+- rabbitmq-graph.gemspec
+homepage: https://github.com/sldblog/rabbitmq-graph
+licenses: []
+metadata:
+  source_code_uri: https://github.com/sldblog/rabbitmq-graph
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.6
+signing_key: 
+specification_version: 4
+summary: Discover RabbitMQ topology
+test_files: []