erd_map 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9b868db53c728786986ef918092b37608399f674c1e0a410fc5f3e9b67d707ad
+  data.tar.gz: 55b585aead0da7727b75e90651ec06dd3636d032cd39078d488169ed20f1374b
+SHA512:
+  metadata.gz: 1ee3f5b8c5d70475d4a6567086c57be4c27e6a1f730f94fb314db64d965621189d43e729c364c08dfcfffde36ac5053513926572e84a295fdae4da2e0c3985ec
+  data.tar.gz: 3067212a98992a87fe1f55787d6a5683a603e0b80a7372bae4456ad6daf332d02d466137aa3503f28b44eebc01cb23153ff53fe1a4fe3921dc2c1a4e3381f23e

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright makicamel
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,106 @@
+# ErdMap
+
+ErdMap is an ERD map viewer as a Rails engine.
+
+Ruby on Rails applications represent their concepts through models. However, since Rails applications often contain numerous models, understanding them can be difficult. ErdMap helps you comprehend your application by visualizing key models and their associations. It provides a clear starting point for understanding the architecture of your application.
+ErdMap initially displays the most "important" models. Then, You can "zoom in" to reveal the next important models interactively, much like navigating a map application.
+
+### Try It Out
+
+Sample visualizations below, based on open-source Rails applications. To try it yourself, open the example HTML files in the `sample` directory.
+
+| [Redmine](https://github.com/redmine/redmine) | [Mastodon](https://github.com/mastodon/mastodon) |
+| ------- | -------- |
+| ![](sample/images/redmine.png) | ![](sample/images/mastdon.png) |
+
+## Dependencies
+
+ErdMap requires Python3 and the following packages: [`networkx`](https://github.com/networkx/networkx), [`bokeh`](https://github.com/bokeh/bokeh), and [`scipy`](https://github.com/scipy/scipy).  
+For Python installation details, refer to the [pyenv installation guide](https://github.com/pyenv/pyenv#installation).
+
+<details><summary>An example for Mac users with Zsh using pyenv for installation</summary>
+
+```bash
+# Install pyenv
+brew install pyenv
+echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
+echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
+echo 'eval "$(pyenv init - zsh)"' >> ~/.zshrc
+
+# Install latest version of python
+pyenv install $(pyenv install --list | grep -E '^\s*[0-9]+\.[0-9]+\.[0-9]+$' | tail -n 1)
+pyenv global $(pyenv install --list | grep -E '^\s*[0-9]+\.[0-9]+\.[0-9]+$' | tail -n 1)
+
+# Install packages using pip
+pip install networkx bokeh scipy
+```
+
+</details>
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem "erd_map", group: [:development]
+```
+
+And then execute:
+
+```bash
+$ bundle
+```
+
+## Usage
+
+Add the following to your `config/routes.rb` and access `/erd_map` in your browser:
+
+```ruby
+Rails.application.routes.draw do
+  mount ErdMap::Engine => "erd_map"
+end
+```
+
+The initial computation might take several seconds. Once completed, the "ErdMap" visualization will be displayed. After the first generation, the map will be cached as an HTML file, so subsequent accesses will display the map instantly without regeneration. If you want to regenerate the map, click the "Re-Compute" button.
+
+The generated HTML file is saved at `/{rails_root}/tmp/erd_map/map.html`.
+
+### Task
+
+You can also explicitly generate the HTML file using rails task.
+
+```bash
+bundle exec rails erd_map
+```
+
+### Map Controls
+
+- Navigation
+  - Wheel Mode: Toggle zooming with the mouse wheel
+  - Zoom In: Reveal more models
+  - Zoom Out: Display fewer models
+
+- Display Options
+  - Tap Mode: Switch between showing associations or communities (see [Algorithm](https://github.com/makicamel/erd_map#Algorithm) section for more about communities)
+  - Display Mode: Toggle between showing only model names or including foreign keys
+
+- Layout
+  - Re-Layout: Randomly rearrange the displayed models
+  - Re-Compute: Regenerate the map to reflect updates to the models
+
+## Algorithm
+
+The initial display shows only the three most "important" models. These models are larger in size, while models displayed upon zooming in are slightly smaller. Importance here is determined by **eigenvector centrality**.
+
+**Eigenvector centrality** is an indicator of how well a model is connected to other highly connected and important models. It considers not just the number of connections a model has, but also the number of important nodes it is connected to.
+
+Additionally, models are organized into groups (communities) and assigned colors for each community. These communities are detected using the **Louvain method**, which discovers strongly connected communities in a network. The method moves and merges nodes iteratively to optimize communities, maximizing modularity (the density of connections), and dividing the network into natural clusters.
+
+Both eigenvector centrality and Louvain method  implementations are provided by [NetworkX](https://github.com/networkx/networkx) library.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/makicamel/erd_map. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/makicamel/erd_map/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,5 @@
+require "bundler/setup"
+
+# load "rails/tasks/statistics.rake"
+
+require "bundler/gem_tasks"

data/app/assets/stylesheets/erd_map/application.css

@@ -0,0 +1,15 @@
+/*
+ * This is a manifest file that'll be compiled into application.css, which will include all the files
+ * listed below.
+ *
+ * Any CSS and SCSS file within this directory, lib/assets/stylesheets, vendor/assets/stylesheets,
+ * or any plugin's vendor/assets/stylesheets directory can be referenced here using a relative path.
+ *
+ * You're free to add application-wide styles to this file and they'll appear at the bottom of the
+ * compiled file so the styles you add here take precedence over styles defined in any other CSS/SCSS
+ * files in this directory. Styles in this file should be added after the last require_* statement.
+ * It is generally better to create a new file per style scope.
+ *
+ *= require_tree .
+ *= require_self
+ */

data/app/controllers/erd_map/application_controller.rb

@@ -0,0 +1,4 @@
+module ErdMap
+  class ApplicationController < ActionController::Base
+  end
+end

data/app/controllers/erd_map/erd_map_controller.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module ErdMap
+  class ErdMapController < ApplicationController
+    FILE_PATH = Rails.root.join("tmp", "erd_map", "map.html")
+
+    def index
+      if File.exist?(FILE_PATH)
+        render html: File.read(FILE_PATH).html_safe
+      else
+        _stdout, stderr, status = Open3.capture3("rails runner 'ErdMap::MapBuilder.build'")
+        if status.success?
+          render html: File.read(FILE_PATH).html_safe
+        else
+          render plain: "Error: #{stderr}", status: :unprocessable_entity
+        end
+      end
+    end
+
+    def update
+      _stdout, stderr, status = Open3.capture3("rails runner 'ErdMap::MapBuilder.build'")
+      if status.success?
+        head :ok
+      else
+        render json: { message: "Error: \n#{stderr}" }, status: :unprocessable_entity
+      end
+    end
+  end
+end

data/app/helpers/erd_map/application_helper.rb

@@ -0,0 +1,4 @@
+module ErdMap
+  module ApplicationHelper
+  end
+end

data/app/models/erd_map/application_record.rb

@@ -0,0 +1,5 @@
+module ErdMap
+  class ApplicationRecord < ActiveRecord::Base
+    self.abstract_class = true
+  end
+end

data/app/views/layouts/erd_map/application.html.erb

@@ -0,0 +1,17 @@
+<!DOCTYPE html>
+<html>
+<head>
+  <title>Erd map</title>
+  <%= csrf_meta_tags %>
+  <%= csp_meta_tag %>
+
+  <%= yield :head %>
+
+  <%= stylesheet_link_tag    "erd_map/application", media: "all" %>
+</head>
+<body>
+
+<%= yield %>
+
+</body>
+</html>

data/config/initializers/erd_map.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+ErdMap.py_call_modules = ErdMap::PyCallModules.new

data/config/routes.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+ErdMap::Engine.routes.draw do
+  root to: "erd_map#index"
+  put "/", to: "erd_map#update"
+end

data/lib/erd_map/engine.rb

@@ -0,0 +1,5 @@
+module ErdMap
+  class Engine < ::Rails::Engine
+    isolate_namespace ErdMap
+  end
+end

data/lib/erd_map/graph.rb

@@ -0,0 +1,225 @@
+# frozen_string_literal: true
+
+module ErdMap
+  class Graph
+    CHUNK_SIZE = 3
+    MAX_COMMUNITY_SIZE = 20
+
+    # @return Array: [{ "NodeA" => [x, y] }, { "NodeA" => [x, y], "NodeB" => [x, y], "NodeC" => [x, y] }, ...]
+    def layouts_by_chunk
+      return @layouts_by_chunk if @layouts_by_chunk
+
+      @layouts_by_chunk = []
+
+      chunked_nodes.each_with_index do |_, i|
+        display_nodes = chunked_nodes[0..i].flatten
+        nodes_size = display_nodes.size
+        k = 1.0 / Math.sqrt(nodes_size) * 3.0
+
+        subgraph = whole_graph.subgraph(display_nodes)
+        layout = nx.spring_layout(subgraph, seed: 1, k: k)
+
+        layout_hash = {}
+        layout.each do |node, xy|
+          layout_hash[node] = [xy[0].to_f, xy[1].to_f]
+        end
+
+        @layouts_by_chunk << layout_hash
+      end
+
+      @layouts_by_chunk
+    end
+
+    # [[nodeA, nodeB, nodeC], [nodeD, nodeE, nodeF, nodeG, ...], ...]
+    def chunked_nodes
+      return @chunked_nodes if @chunked_nodes
+
+      centralities = nx.eigenvector_centrality(whole_graph) # { node_name => centrality }
+      sorted_nodes = centralities.sort_by { |_node, centrality| centrality }.reverse.map(&:first)
+
+      chunk_sizes = []
+      total_nodes = sorted_nodes.size
+      while chunk_sizes.sum < total_nodes
+        chunk_sizes << (CHUNK_SIZE ** (chunk_sizes.size + 1))
+      end
+
+      offset = 0
+      @chunked_nodes = chunk_sizes.each_with_object([]) do |size, nodes|
+        slice = sorted_nodes[offset, size]
+        break nodes if slice.nil? || slice.empty?
+        offset += size
+        nodes << slice
+      end
+    end
+
+    # @return Hash: { String: Integer }
+    def node_with_community_index
+      return @node_with_community_index if @node_with_community_index
+
+      whole_communities = networkx_community.louvain_communities(whole_graph).map { |communities| PyCall::List.new(communities).to_a }
+      communities = split_communities(whole_graph, whole_communities)
+
+      @node_with_community_index = {}
+      communities.each_with_index do |community, i|
+        community.each do |node_name|
+          @node_with_community_index[node_name] = i
+        end
+      end
+      @node_with_community_index
+    end
+
+    def initial_nodes
+      chunked_nodes.first
+    end
+
+    def initial_layout
+      layouts_by_chunk.first
+    end
+
+    def whole_layout
+      layouts_by_chunk.last
+    end
+
+    def node_names
+      @node_names ||= PyCall::List.new(whole_graph.nodes)
+    end
+
+    def edges
+      @edges ||= PyCall::List.new(whole_graph.edges)
+    end
+
+    def node_radius
+      @node_radius ||= node_names.map { |node_name| nodes_with_radius_according_to_chunk_index[node_name] }
+    end
+
+    def connections
+      @connections ||= edges.each_with_object(Hash.new { |hash, key| hash[key] = [] }) do |(a, b), hash|
+        hash[a] << b
+        hash[b] << a
+      end
+    end
+
+    def association_columns
+      return @association_columns if @association_columns
+
+      @association_columns = Hash.new { |hash, key| hash[key] = [] }
+      whole_models.each do |model|
+        model.reflect_on_all_associations(:belongs_to).select { |mod| !mod.options[:polymorphic] }.map do |target|
+          if target.try(:foreign_key) && model.column_names.include?(target.foreign_key)
+            @association_columns[model.name] << target.foreign_key
+          end
+        end
+      end
+      @association_columns
+    end
+
+    def node_colors
+      return @node_colors if @node_colors
+
+      palette = [
+        "#a6cee3", "#1f78b4", "#b2df8a", "#33a02c", "#fb9a99",
+        "#e74446", "#fdbf6f", "#ff7f00", "#cab2d6", "#7850a4",
+        "#ffff99", "#b8693d", "#8dd3c7", "#ffffb3", "#bebada",
+        "#fb8072", "#80b1d3", "#fdb462", "#b3de69", "#fccde5",
+        "#d9d9d9", "#bc80bd", "#ccebc5", "#ffed6f", "#1b9e77",
+        "#d95f02", "#7570b3", "#ef73b2", "#66a61e", "#e6ab02"
+      ]
+      community_map = node_with_community_index
+      @node_colors = node_names.map do |node_name|
+        community_index = community_map[node_name]
+        palette[community_index % palette.size]
+      end
+    end
+
+    private
+
+    attr_reader :nx, :networkx_community
+    attr_reader :whole_graph
+
+    def initialize
+      import_modules = ErdMap.py_call_modules.imported_modules
+      @nx = import_modules[:nx]
+      @networkx_community = import_modules[:networkx_community]
+      @whole_graph = build_whole_graph
+    end
+
+    def whole_models
+      Rails.application.eager_load!
+      @whole_models ||= ActiveRecord::Base.descendants
+        .reject { |model| model.name.in?(%w[ActiveRecord::SchemaMigration ActiveRecord::InternalMetadata]) }
+        .select(&:table_exists?)
+    end
+
+    def build_whole_graph
+      whole_graph = nx.Graph.new
+
+      whole_models.each do |model|
+        whole_graph.add_node(model.name)
+        [:has_many, :has_one, :belongs_to].each do |association_type|
+          model
+            .reflect_on_all_associations(association_type)
+            .select { |mod| !mod.options[:polymorphic] && !mod.options[:anonymous_class] }
+            .map(&:class_name)
+            .uniq
+            .select { |target| target.constantize.respond_to?(:column_names) }
+            .map do |target|
+            if association_type == :belongs_to
+              whole_graph.add_edge(target, model.name)
+            else
+              whole_graph.add_edge(model.name, target)
+            end
+          end
+        end
+      end
+      whole_graph
+    end
+
+    # { "NodeA" => 0, "NodeB" => 0, "NodeC" => 1, ... }
+    def nodes_with_chunk_index
+      return @nodes_with_chunk_index if @nodes_with_chunk_index
+      @nodes_with_chunk_index = {}
+      chunked_nodes.each_with_index do |chunk, i|
+        chunk.each { |node_name| @nodes_with_chunk_index[node_name] = i }
+      end
+      @nodes_with_chunk_index
+    end
+
+    def nodes_with_radius_according_to_chunk_index
+      return @nodes_with_radius_according_to_chunk_index if @nodes_with_radius_according_to_chunk_index
+
+      max_node_size = 60
+      min_node_size = 20
+      node_size_step = 10
+
+      @nodes_with_radius_according_to_chunk_index = {}
+      chunked_nodes.each_with_index do |chunk, chunk_index|
+        chunk.each do |node_name|
+          size = max_node_size - (chunk_index * node_size_step)
+          @nodes_with_radius_according_to_chunk_index[node_name] = (size < min_node_size) ? min_node_size : size
+        end
+      end
+      @nodes_with_radius_according_to_chunk_index
+    end
+
+    def split_communities(graph, communities)
+      result = []
+
+      communities.each do |community|
+        if community.size <= MAX_COMMUNITY_SIZE
+          result << community
+        else
+          subgraph = graph.subgraph(community)
+          sub_communities = networkx_community.louvain_communities(subgraph).map { |comm| PyCall::List.new(comm).to_a }
+          if sub_communities.size == 1 && (sub_communities[0] - community).empty?
+            result << community
+          else
+            splitted_sub = split_communities(subgraph, sub_communities)
+            result.concat(splitted_sub)
+          end
+        end
+      end
+
+      result
+    end
+  end
+end