typed_dag 1.0.1 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e47a0669cc0a0487993711a18e1a37df86b809a5
-  data.tar.gz: 11f39e3b2f2a189bb28d7f93d6ff4f3b309d6584
+  metadata.gz: a7e31c0c6a123f535ad41b59c8aabaac96129a70
+  data.tar.gz: 78ee741a3bf16ea7014f2cf69a8a3f5903c9fab1
 SHA512:
-  metadata.gz: c11feb64fa4401c79b9285c2629b03d1e7558c6e8ef6f1202b258726beb9b464c9d406ee0bf23a14ed713260f2cbf49e56c630ade313e38fd4f39f2a06e4b57c
-  data.tar.gz: 5662fb1a45f66349acc32697f56b09b3e292b9096c17b43ecd94c068203d164e62e569298120cad2e320a312de1f6a1b13da45f5fac1f4389f602d11446998d2
+  metadata.gz: 7941c61304bd469a8bff392405f09719f7246e6817e991cf835f7435d0570d8b832b37edaf87186ab359b57a20a0f5a5e949ffb65f87149b6745b4ab21be3b3c
+  data.tar.gz: 206c14332f5e58353063cf45ac0e8d279f632f017a14bbd1c96a0a2f4ab8e9cf3424f33cc1b2378c13eaf9363259caf2a8b26e920f37ec4a7eea412e24f2e383

data/README.md

@@ -35,6 +35,10 @@ Again using the forum application as an example, one would have messages in a hi
 
 Please note that the name of the scopes (e.g. `children`, `hierarchy_roots` and `referenced`) and constants (`Message`) have to be configured.
 
+## Requirements
+
+* Rails >= 5.0
+* MySQL or PostgreSQL (>= 9.5) as `UPSERT` statements are used
 
 ## Installation
 
@@ -62,10 +66,12 @@ To avoid having to configure TypedDag twice, in the node and in the edge AR mode
 
 ```
   # /config/initializers/typed_dag.rb
+  # configuration for Relation/Message
   TypedDag::Configuration.set edge_class_name: 'Relation',
                               node_class_name: 'Message',
                               from_column: 'ancestor_id',
                               to_column: 'descendant_id',
+                              count_column: 'amount',
                               types: { hierarchy: { from: { name: :parent, limit: 1 },
                                                     to: :children,
                                                     all_from: :ancestors,
@@ -75,6 +81,8 @@ To avoid having to configure TypedDag twice, in the node and in the edge AR mode
                                                      all_from: :all_invalidated_by,
                                                      all_to: :all_invalidates } }
 
+
+  # unrelated configuration for Edge/Node
   TypedDag::Configuration.set edge_class_name: 'Edge'
                               node_class_name: 'Node',
                               types: { edge: { from: :edges_from,
@@ -93,6 +101,7 @@ The following options exist:
  * `node_class_name`: The name of the AR model whose instances serve as the nodes of the dag
  * `from_column` (default `from_id`): The name of the column in the edges AR model that refer to the node the edge starts from
  * `to_column` (default `to_id`): The name of the column in the edges AR model that refer to the node the edge ends in
+ * `count_column` (default `count`): The name of the column in the edges AR model that keeps track of the number of identical edges between from and to
  * `types`: The hash of type configurations. The key of each configuration will need to be present as a column in the edge's DB table.
   * `from`: The AR association's name for nodes having a relation which end in the current node, have the type specified by the key and are not transitive (have only one hop). Only for `from` can one specify a limit to the number of relations a node can have. Doing this turns the DAG into a tree which is usefull for hierarchies. If a limit needs to be specified, the configuration has to be provided as `{ name: [association's name], limit: 1 }`. If no limit is given, the association's name can be provided as a symbol.
   * `to`: The AR association's name for nodes having a relation which start from the current node, have the type specified by the key and are not transitive (have only one hop)
@@ -156,7 +165,8 @@ The edge's table needs to be created containing at least the following columns (
 
  * `from_id`: A reference to the node the edge starts from.
  * `to_id`: A reference to the node the edge ends in.
- * `[key]`: A column of type integer for every type the dag is to support
+ * `count`: The counter column for the number of similar (transitive) edges between to and from.
+ * `[key]`: A column of type integer for every type the dag is to support.
 
 A migration to create such a table could look like this:
 
@@ -166,6 +176,8 @@ A migration to create such a table could look like this:
       t.references :from, null: false
       t.references :to, null: false
 
+      t.column :count, :integer, null: false, default: 0
+
       t.column :hierarchy, :integer, null: false, default: 0
       t.column :reference, :integer, null: false, default: 0
     end
@@ -173,17 +185,20 @@ A migration to create such a table could look like this:
     add_foreign_key :edges, :nodes, column: :from_id
     add_foreign_key :edges, :nodes, column: :to_id
 
+    # give the index a custom name to avoid running into length limitation when having a couple of columns
+    # in the index
     add_index :edges, [:hierarchy, :reference], name: `index_on_type_columns`
+    add_index :edges, :count, where: 'count = 0'
   end
 ```
 
-the table can also have additional columns. They will not interfere with TypedDag.
+The table can also have additional columns. They will not interfere with TypedDag.
 
-Which indices to use will depend on the data added but having an index over all the type columns is a good start.
+Which indices to use will depend on the data added but having an index over all the type columns is a good start. A partial index on count speeds up deleting edges while also being very lightweight to maintain.
 
-There are no requirements for the node's table.
+There are no requirements on the node's table.
 
-When migrating from an different library, the details of course depend on the library used. If it is one of the many having a `parent_id` column on the node table, one would first have to create the edge table as outlined above and then add a SQL statement like this:
+When migrating from a different library, the details of course depend on the library used. If it is one of the many having a `parent_id` column on the node table, one would first have to create the edge table as outlined above and then add a SQL statement like this:
 
 ```
   ActiveRecord::Base.connection.execute <<-SQL

data/lib/typed_dag/configuration.rb

@@ -56,6 +56,10 @@ class TypedDag::Configuration
     config[:to_column] || 'to_id'
   end
 
+  def count_column
+    config[:count_column] || 'count'
+  end
+
   def types
     config[:types] || default_types
   end

data/lib/typed_dag/edge/closure_maintenance.rb

@@ -3,12 +3,17 @@ module TypedDag::Edge
     extend ActiveSupport::Concern
 
     included do
+      before_create :set_count
       after_create :add_closures
       after_update :alter_closure
       after_destroy :truncate_closures
 
       private
 
+      def set_count
+        send("#{_dag_options.count_column}=", 1) if send(_dag_options.count_column).zero?
+      end
+
       def add_closures
         return unless direct?
 
@@ -20,7 +25,7 @@ module TypedDag::Edge
         # However, #persisted? will be false for destroyed records.
         return unless direct? && !new_record?
 
-        self.class.connection.execute truncate_dag_closure_sql(self)
+        update_and_delete_closure(self)
       end
 
       def truncate_closures_with_former_values
@@ -35,7 +40,7 @@ module TypedDag::Edge
 
         former_values_relation.attributes = changes
 
-        self.class.connection.execute truncate_dag_closure_sql(former_values_relation)
+        update_and_delete_closure(former_values_relation)
       end
 
       def alter_closure
@@ -45,6 +50,11 @@ module TypedDag::Edge
         add_closures
       end
 
+      def update_and_delete_closure(relation)
+        self.class.connection.execute truncate_dag_closure_sql(relation)
+        self.class.connection.execute delete_zero_count_sql(relation)
+      end
+
       def add_dag_closure_sql
         TypedDag::Sql::AddClosure.sql(self)
       end
@@ -53,6 +63,10 @@ module TypedDag::Edge
         TypedDag::Sql::TruncateClosure.sql(relation)
       end
 
+      def delete_zero_count_sql(relation)
+        TypedDag::Sql::DeleteZeroCount.sql(relation)
+      end
+
       def from_id_value
         send(_dag_options.from_column)
       end

data/lib/typed_dag/sql.rb

@@ -6,5 +6,6 @@ module TypedDag
     require 'typed_dag/sql/get_circular'
     require 'typed_dag/sql/remove_invalid_relation'
     require 'typed_dag/sql/insert_reflexive'
+    require 'typed_dag/sql/delete_zero_count'
   end
 end

data/lib/typed_dag/sql/add_closure.rb

@@ -15,18 +15,49 @@ module TypedDag::Sql::AddClosure
 
     def sql
       <<-SQL
-        INSERT INTO #{table_name}
-          (#{from_column},
-          #{to_column},
-          #{type_select_list})
-        #{closure_select}
+        #{insert_sql}
+        #{on_duplicate}
       SQL
     end
 
     private
 
+    def on_duplicate
+      if helper.mysql_db?
+        on_duplicate_mysql
+      else
+        on_duplicate_postgresql
+      end
+    end
+
+    def on_duplicate_mysql
+      <<-SQL
+        ON DUPLICATE KEY
+        UPDATE #{count_column} = #{table_name}.#{count_column} + VALUES(#{count_column})
+      SQL
+    end
+
+    def on_duplicate_postgresql
+      <<-SQL
+        ON CONFLICT (#{column_list})
+        DO UPDATE SET #{count_column} = #{table_name}.#{count_column} + EXCLUDED.#{count_column}
+      SQL
+    end
+
+    def insert_sql
+      <<-SQL
+        INSERT INTO #{table_name}
+          (#{column_list}, #{count_column})
+        #{closure_select}
+      SQL
+    end
+
     def closure_select
       TypedDag::Sql::SelectClosure.sql(relation)
     end
+
+    def column_list
+      "#{from_column}, #{to_column}, #{type_select_list}"
+    end
   end
 end

data/lib/typed_dag/sql/delete_zero_count.rb

@@ -0,0 +1,22 @@
+require 'typed_dag/sql/helper'
+
+module TypedDag::Sql::DeleteZeroCount
+  def self.sql(relation)
+    Sql.new(relation).sql
+  end
+
+  class Sql
+    include TypedDag::Sql::RelationAccess
+
+    def initialize(relation)
+      self.relation = relation
+    end
+
+    def sql
+      <<-SQL
+        DELETE FROM #{table_name}
+        WHERE #{count_column} = 0
+      SQL
+    end
+  end
+end

data/lib/typed_dag/sql/get_circular.rb

@@ -13,8 +13,10 @@ module TypedDag::Sql::GetCircular
     def sql(depth)
       <<-SQL
         SELECT
-          r1.#{helper.from_column},
-          r1.#{helper.to_column}
+          r1.#{helper.from_column} AS r1_from_column,
+          r1.#{helper.to_column} AS r1_to_column,
+          r2.#{helper.from_column} AS r2_from_column,
+          r2.#{helper.to_column} AS r2_to_column
         FROM #{helper.table_name} r1
         JOIN #{helper.table_name} r2
         ON #{join_condition(depth)}

data/lib/typed_dag/sql/helper.rb

@@ -23,13 +23,23 @@ class TypedDag::Sql::Helper
     configuration.type_columns
   end
 
+  def count_column
+    configuration.count_column
+  end
+
   def type_select_list
     type_columns.join(', ')
   end
 
   def type_select_summed_columns(prefix1, prefix2)
     type_columns
-      .map { |column| "#{prefix1}.#{column} + #{prefix2}.#{column}" }
+      .map { |column| "#{prefix1}.#{column} + #{prefix2}.#{column} " }
+      .join(', ')
+  end
+
+  def type_select_summed_columns_aliased(prefix1, prefix2)
+    type_columns
+      .map { |column| "(#{prefix1}.#{column} + #{prefix2}.#{column}) #{column}" }
       .join(', ')
   end
 
@@ -45,6 +55,10 @@ class TypedDag::Sql::Helper
     type_columns.map { |column| "#{prefix}#{column} = 1" }.join(' XOR ')
   end
 
+  def mysql_db?
+    ActiveRecord::Base.connection.adapter_name == 'Mysql2'
+  end
+
   private
 
   attr_accessor :configuration

data/lib/typed_dag/sql/insert_closure_of_depth.rb

@@ -11,19 +11,50 @@ module TypedDag::Sql::InsertClosureOfDepth
     end
 
     def sql(depth)
+      if helper.mysql_db?
+        sql_mysql(depth)
+      else
+        sql_postgresql(depth)
+      end
+    end
+
+    private
+
+    def sql_mysql(depth)
+      <<-SQL
+        #{insert_sql(depth)}
+        ON DUPLICATE KEY
+        UPDATE #{helper.table_name}.#{helper.count_column} = #{helper.table_name}.#{helper.count_column} + VALUES(#{helper.count_column})
+      SQL
+    end
+
+    def sql_postgresql(depth)
+      <<-SQL
+        #{insert_sql(depth)}
+        ON CONFLICT (#{insert_list})
+        DO UPDATE SET #{helper.count_column} = #{helper.table_name}.#{helper.count_column} + EXCLUDED.#{helper.count_column}
+      SQL
+    end
+
+    def insert_sql(depth)
       <<-SQL
         INSERT INTO #{helper.table_name}
-          (#{insert_list})
-        SELECT
-          #{select_list}
+          (#{insert_list}, #{helper.count_column})
+        SELECT #{insert_list}, #{helper.count_column} FROM
+          (#{sum_select(depth)}) to_insert
+      SQL
+    end
+
+    def sum_select(depth)
+      <<-SQL
+        SELECT #{select_list}, SUM(r1.#{helper.count_column} * r2.#{helper.count_column}) AS #{helper.count_column}
         FROM #{helper.table_name} r1
         JOIN #{helper.table_name} r2
         ON #{join_condition(depth)}
+        GROUP BY #{group_list}
       SQL
     end
 
-    private
-
     def insert_list
       [helper.from_column,
        helper.to_column,
@@ -31,6 +62,14 @@ module TypedDag::Sql::InsertClosureOfDepth
     end
 
     def select_list
+      <<-SQL
+        r1.#{helper.from_column},
+        r2.#{helper.to_column},
+        #{helper.type_select_summed_columns_aliased('r1', 'r2')}
+      SQL
+    end
+
+    def group_list
       <<-SQL
         r1.#{helper.from_column},
         r2.#{helper.to_column},

data/lib/typed_dag/sql/insert_reflexive.rb

@@ -14,8 +14,9 @@ module TypedDag::Sql::InsertReflexive
       <<-SQL
         INSERT INTO #{helper.table_name}
           (#{helper.from_column},
-           #{helper.to_column})
-        SELECT id, id
+           #{helper.to_column},
+           #{helper.count_column})
+        SELECT id, id, 1
         FROM #{helper.node_table_name}
       SQL
     end

data/lib/typed_dag/sql/relation_access.rb

@@ -11,6 +11,7 @@ module TypedDag::Sql::RelationAccess
     delegate :table_name,
              :from_column,
              :to_column,
+             :count_column,
              :type_columns,
              :type_select_list,
              to: :helper

data/lib/typed_dag/sql/select_closure.rb

@@ -15,15 +15,26 @@ module TypedDag::Sql::SelectClosure
     def sql
       <<-SQL
         SELECT
-          r1.#{from_column},
-          r2.#{to_column},
-          #{depth_sum_case}
+          #{from_column},
+          #{to_column},
+          #{type_columns.join(', ')},
+          SUM(#{count_column}) AS #{count_column}
         FROM
-          #{table_name} r1
-        JOIN
-          #{table_name} r2
-        ON
-          (#{relations_join_combines_paths_condition})
+          (SELECT
+            r1.#{from_column},
+            r2.#{to_column},
+            #{depth_sum_case},
+            r1.#{count_column} * r2.#{count_column} AS #{count_column}
+          FROM
+            #{table_name} r1
+          JOIN
+            #{table_name} r2
+          ON
+            (#{relations_join_combines_paths_condition})) unique_rows
+        GROUP BY
+          #{from_column},
+          #{to_column},
+          #{type_columns.join(', ')}
       SQL
     end
 
@@ -40,7 +51,7 @@ module TypedDag::Sql::SelectClosure
             ELSE 0
             END AS #{column}
         SQL
-      end.join(', ')
+      end.map(&:strip).join(', ')
     end
 
     def relations_join_combines_paths_condition

data/lib/typed_dag/sql/truncate_closure.rb

@@ -13,7 +13,7 @@ module TypedDag::Sql::TruncateClosure
     end
 
     def sql
-      if mysql_db?
+      if helper.mysql_db?
         sql_mysql
       else
         sql_postgresql
@@ -26,39 +26,37 @@ module TypedDag::Sql::TruncateClosure
 
     def sql_mysql
       <<-SQL
-        DELETE
-          deletion_table
-        FROM
-          #{table_name} deletion_table
-        INNER JOIN
-          #{selection_table} selection_table
-        ON deletion_table.id = selection_table.id
+        UPDATE #{table_name}
+        JOIN
+          (#{closure_select}) removed_#{table_name}
+        ON #{table_name}.#{from_column} = removed_#{table_name}.#{from_column}
+        AND #{table_name}.#{to_column} = removed_#{table_name}.#{to_column}
+        AND #{types_equality_condition}
+        SET
+          #{table_name}.#{count_column} = #{table_name}.#{count_column} - removed_#{table_name}.#{count_column}
       SQL
     end
 
     def sql_postgresql
       <<-SQL
-        DELETE FROM
-          #{table_name} deletion_table
-        USING
-          #{selection_table} selection_table
-        WHERE deletion_table.id = selection_table.id
+        UPDATE #{table_name}
+        SET
+          #{count_column} = #{table_name}.#{count_column} - removed_#{table_name}.#{count_column}
+        FROM
+          (#{closure_select}) removed_#{table_name}
+        WHERE #{table_name}.#{from_column} = removed_#{table_name}.#{from_column}
+        AND #{table_name}.#{to_column} = removed_#{table_name}.#{to_column}
+        AND #{types_equality_condition}
       SQL
     end
 
     def selection_table
       <<-SQL
-          (SELECT id
-          FROM (
-            SELECT COUNT(*) count, #{from_column}, #{to_column}, #{type_select_list}
-            FROM
-              (#{closure_select}) aggregation
-            GROUP BY #{from_column}, #{to_column}, #{type_select_list}) criteria
-
-          JOIN
-            (#{rank_similar_relations}) ranked
-          ON
-            #{ranked_critieria_join_condition})
+        (
+         SELECT COUNT(*) #{count_column}, #{from_column}, #{to_column}, #{type_select_list}
+         FROM
+           (#{closure_select}) aggregation
+         GROUP BY #{from_column}, #{to_column}, #{type_select_list})
       SQL
     end
 
@@ -66,113 +64,10 @@ module TypedDag::Sql::TruncateClosure
       TypedDag::Sql::SelectClosure.sql(relation)
     end
 
-    def rank_similar_relations
-      if mysql_db?
-        rank_similar_relations_mysql
-      else
-        rank_similar_relations_postgresql
-      end
-    end
-
-    def ranked_critieria_join_condition
-      <<-SQL
-        ranked.#{from_column} = criteria.#{from_column}
-        AND ranked.#{to_column} = criteria.#{to_column}
-        AND #{types_equality_condition}
-        AND count >= row_number
-      SQL
-    end
-
-    def type_column_values_pairs
-      type_columns.map do |column|
-        [column, relation.send(column)]
-      end
-    end
-
     def types_equality_condition
       type_columns.map do |column|
-        "ranked.#{column} = criteria.#{column}"
+        "#{table_name}.#{column} = removed_#{table_name}.#{column}"
       end.join(' AND ')
     end
-
-    def mysql_db?
-      ActiveRecord::Base.connection.adapter_name == 'Mysql2'
-    end
-
-    def rank_similar_relations_mysql
-      <<-SQL
-        SELECT
-          id,
-          #{from_column},
-          #{to_column},
-          #{type_select_list},
-          greatest(@cur_count := IF(#{compare_mysql_variables},
-                                 @cur_count + 1, 1),
-                   least(0, #{assign_mysql_variables})) AS row_number
-        FROM
-          #{table_name}
-
-        CROSS JOIN (SELECT #{initialize_mysql_variables}) params_initialization
-
-        WHERE
-          #{only_relations_in_closure_condition}
-
-        ORDER BY #{from_column}, #{to_column}, #{type_select_list}
-      SQL
-    end
-
-    def rank_similar_relations_postgresql
-      <<-SQL
-        SELECT *, ROW_NUMBER() OVER(
-          PARTITION BY #{from_column}, #{to_column}, #{type_select_list}
-        )
-        FROM
-          #{table_name}
-        WHERE
-          #{only_relations_in_closure_condition}
-      SQL
-    end
-
-    def only_relations_in_closure_condition
-      <<-SQL
-        #{from_column} IN (SELECT #{from_column} FROM #{table_name} WHERE #{to_column} = #{from_id_value}) OR #{from_column} = #{from_id_value}
-      AND
-        #{to_column} IN (SELECT #{to_column} FROM #{table_name} WHERE #{from_column} = #{from_id_value})
-      SQL
-    end
-
-    def initialize_mysql_variables
-      variable_string = "@cur_count := NULL,
-                         @cur_#{from_column} := NULL,
-                         @cur_#{to_column} := NULL"
-
-      type_columns.each do |column|
-        variable_string += ", @cur_#{column} := NULL"
-      end
-
-      variable_string
-    end
-
-    def assign_mysql_variables
-      variable_string = "@cur_#{from_column} := #{from_column},
-                         @cur_#{to_column} := #{to_column}"
-
-      type_columns.each do |column|
-        variable_string += ", @cur_#{column} := #{column}"
-      end
-
-      variable_string
-    end
-
-    def compare_mysql_variables
-      variable_string = "@cur_#{from_column} = #{from_column} AND
-                         @cur_#{to_column} = #{to_column}"
-
-      type_columns.each do |column|
-        variable_string += " AND @cur_#{column} = #{column}"
-      end
-
-      variable_string
-    end
   end
 end