samvera-nesting_indexer 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 3933e26eda523a5609b140bccf34d4c291669292
-  data.tar.gz: d6d13b4ba28d3863aaaf32f40a8524de8f37ab5b
+  metadata.gz: 37f64367d992a4d794752c4eda559a3e066933f3
+  data.tar.gz: 5de4fc1861c009b962fecd391582bc2240f329b7
 SHA512:
-  metadata.gz: fbd5fa597ab1afc270d34a8e501e39a9aa6d066f3442b87aa7f692c9bbbd8d719dcd6c08cd196e0b95a17dc13918588b33fad42b8f86e4cb65d0e25d31297f8b
-  data.tar.gz: 164221f096a42d550d624f6658fbc0e3127560935f05684f573c53de5c7772b4b68665a5058788a32fda61c06fbaf41936da0ea3bb32505f0cc874b7a9b1b3ec
+  metadata.gz: dad5de0e38c405e321e7f520cb0c44137423fa942e81218ba809e4dc9dc12255b15d69edc59f1c2b8aaa866504da064c1df1d45cd059fcf3e5069f234b2d2990
+  data.tar.gz: 6882d1ea6b0a40a9c738a420efe54ec3a984c591d16d20247bd3135f5a919883f249e13a3965585a7be9a6793e0ee5d1e83b656b8a858f832b71f7f14e195f6c

data/README.md

@@ -18,6 +18,10 @@ The Samvera::NestingIndexer gem is responsible for indexing the graph relationsh
 
 This is a sandbox to work through the reindexing strategy as it relates to [CurateND Collections](https://github.com/ndlib/samvera_nd/issues/420). At this point the code is separate to allow for raid testing and prototyping (no sense spinning up SOLR and Fedora to walk an arbitrary graph).
 
+### Notation
+
+When B is a member of A, I am using the `A ={ B` notation. When C is a member of B and B is a member of A, I'll chain these together `A ={ B ={ C`.
+
 ## Concepts
 
 As we are indexing objects, we have two types of documents:
@@ -36,6 +40,8 @@ See [Samvera::NestingIndexer::Documents::IndexDocument](./lib/samvera/nesting_in
 
 To reindex a single document, we leverage the [`Samvera::NestingIndexer.reindex_relationships`](./lib/samvera/nesting_indexer.rb) method.
 
+To reindex all of the documents, we leverage the [`Samvera::NestingIndexer.reindex_all!`](lib/samvera/nesting_indexer.rb) method. **Warning: This is a very slow process.**
+
 ## Examples
 
 Given the following PreservationDocuments:
@@ -87,10 +93,7 @@ RSpec.describe MyCustomAdapter
 end
 ```
 
-
-
-
-[See CurateND for our adaptor configuration](https://github.com/ndlib/samvera_nd/blob/6fbe79c9725c0f8b4641981044ec250c5163053b/config/initializers/samvera_config.rb#L32-L35).
+[See CurateND for Notre Dame's adaptor configuration](https://github.com/ndlib/samvera_nd/blob/6fbe79c9725c0f8b4641981044ec250c5163053b/config/initializers/samvera_config.rb#L32-L35).
 
 ## Considerations
 
@@ -100,3 +103,21 @@ Given a single object A, when we reindex A, we:
 * Iterate through each descendant, in a breadth-first process, to reindex it (and each descendant's descendants).
 
 This is a potentially time consumptive process and should not be run within the request cycle.
+
+### Cycle Detections
+
+When dealing with nested graphs, there is a danger of creating an cycle (e.g. `A ={ B ={ A`). Samvera::NestingIndexer implements two guards to short-circuit the indexing of cyclic graphs:
+
+* Enforcing a maximum nesting depth of the graph
+* Checking that an object is not its own ancestor (`Samvera::NestingIndexer::RelationshipReindexer#guard_against_possiblity_of_self_ancestry`)
+
+The [`./spec/features/reindex_pid_and_descendants_spec.rb`](spec/features/reindex_pid_and_descendants_spec.rb) contains examples of behavior.
+
+**NOTE: These guards to prevent indexing cyclic graphs do not prevent the underlying preservation document from creating its own cyclic graph.**
+
+## TODO
+
+- [ ] Incorporate additional logging
+- [ ] Build methods to allow for fanning out the reindexing. At present, when we reindex a node and its "children", we run that entire process within a single context. Likewise, we run a single process when reindexing EVERYTHING.
+- [ ] Promote from [samvera-labs](https://github.com/samvera-labs) to [samvera](https://github.com/samvera) via the [promotion process](http://samvera-labs.github.io/promotion.html).
+- [ ] Write adapter method to assist in guarding against self-ancestry. We could probably expose a base adapter that has the method through use of the other adapter methods.

data/lib/samvera/nesting_indexer.rb

@@ -13,11 +13,14 @@ module Samvera
     # In a perfect world we could reindex the id as well; But that is for another test.
     #
     # @param id [String] - The permanent identifier of the object that will be reindexed along with its children.
-    # @param maximum_nesting_depth [Integer] - there to guard against cyclical graphs
+    # @param maximum_nesting_depth [Integer] - used to short-circuit overly deep nesting as well as prevent accidental cyclic graphs
+    #                                          from creating an infinite loop.
     # @return [Boolean] - It was successful
-    # @raise Samvera::Exceptions::CycleDetectionError - A potential cycle was detected
+    # @raise Samvera::Exceptions::CycleDetectionError - A possible cycle was detected
+    # @raise Samvera::Exceptions::ExceededMaximumNestingDepthError - We exceeded our maximum depth
+    # @raise Samvera::Exceptions::DocumentIsItsOwnAncestorError - A document we were about to index appeared to be its own ancestor
     def self.reindex_relationships(id:, maximum_nesting_depth: configuration.maximum_nesting_depth)
-      RelationshipReindexer.call(id: id, maximum_nesting_depth: maximum_nesting_depth, adapter: adapter)
+      RelationshipReindexer.call(id: id, maximum_nesting_depth: maximum_nesting_depth, configuration: configuration)
       true
     end
 
@@ -29,14 +32,14 @@ module Samvera
 
     # @api public
     # Responsible for reindexing the entire preservation layer.
-    # @param maximum_nesting_depth [Integer] - there to guard against cyclical graphs
+    # @param maximum_nesting_depth [Integer] - there to guard against cyclic graphs
     # @return [Boolean] - It was successful
-    # @raise Samvera::Exceptions::CycleDetectionError - A potential cycle was detected
+    # @raise Samvera::Exceptions::ReindexingError - There was a problem reindexing the graph.
     def self.reindex_all!(maximum_nesting_depth: configuration.maximum_nesting_depth)
       # While the RepositoryReindexer is responsible for reindexing everything, I
       # want to inject the lambda that will reindex a single item.
       id_reindexer = method(:reindex_relationships)
-      RepositoryReindexer.call(maximum_nesting_depth: maximum_nesting_depth, id_reindexer: id_reindexer, adapter: adapter)
+      RepositoryReindexer.call(maximum_nesting_depth: maximum_nesting_depth, id_reindexer: id_reindexer, configuration: configuration)
       true
     end
 

data/lib/samvera/nesting_indexer/adapters/in_memory_adapter.rb

@@ -88,7 +88,7 @@ module Samvera
           end
 
           def find_each
-            cache.each { |_key, document| yield(document) }
+            cache.each_value { |document| yield(document) }
           end
 
           def clear_cache!

data/lib/samvera/nesting_indexer/adapters/interface_behavior_spec.rb

@@ -61,7 +61,7 @@ if defined?(RSpec)
       end
 
       it 'expects a block' do
-        expect(block_parameter_extracter.call(subject)).to be_present
+        expect(block_parameter_extracter.call(subject)).to eq([:block])
       end
     end
     describe '.each_child_document_of' do
@@ -76,13 +76,13 @@ if defined?(RSpec)
       end
 
       it 'expects a block' do
-        expect(block_parameter_extracter.call(subject)).to be_present
+        expect(block_parameter_extracter.call(subject)).to eq([:block])
       end
     end
     describe '.write_document_attributes_to_index_layer' do
       subject { described_class.method(:write_document_attributes_to_index_layer) }
 
-      it 'requires the :attributes keyword (and does not require any others)' do
+      it 'requires the :ancestors, :id, :parent_ids, and :pathnames keyword (and does not require any others)' do
         expect(required_keyword_parameters.call(subject)).to eq(%i(ancestors id parent_ids pathnames))
       end
 

data/lib/samvera/nesting_indexer/configuration.rb

@@ -1,5 +1,6 @@
 require 'samvera/nesting_indexer/adapters/abstract_adapter'
 require 'samvera/nesting_indexer/exceptions'
+require 'logger'
 
 module Samvera
   # :nodoc:
@@ -9,11 +10,14 @@ module Samvera
     class Configuration
       DEFAULT_MAXIMUM_NESTING_DEPTH = 15
 
-      def initialize(maximum_nesting_depth: DEFAULT_MAXIMUM_NESTING_DEPTH)
+      def initialize(maximum_nesting_depth: DEFAULT_MAXIMUM_NESTING_DEPTH, logger: default_logger)
         self.maximum_nesting_depth = maximum_nesting_depth
+        self.logger = logger
       end
 
-      attr_reader :maximum_nesting_depth
+      attr_reader :maximum_nesting_depth, :logger
+
+      attr_writer :logger
 
       def maximum_nesting_depth=(input)
         @maximum_nesting_depth = input.to_i
@@ -68,6 +72,14 @@ module Samvera
         require 'samvera/nesting_indexer/adapters/in_memory_adapter'
         Adapters::InMemoryAdapter
       end
+
+      def default_logger
+        if defined?(Rails.logger)
+          Rails.logger
+        else
+          Logger.new($stdout)
+        end
+      end
     end
     private_constant :Configuration
   end

data/lib/samvera/nesting_indexer/documents.rb

@@ -3,6 +3,8 @@ require 'dry-equalizer'
 module Samvera
   module NestingIndexer
     module Documents
+      ANCESTOR_AND_PATHNAME_DELIMITER = '/'.freeze
+
       # @api public
       #
       # A simplified document that reflects the necessary attributes for re-indexing

data/lib/samvera/nesting_indexer/exceptions.rb

@@ -30,9 +30,34 @@ module Samvera
       # Raised when we may have detected a cycle within the graph
       class CycleDetectionError < RuntimeError
         attr_reader :id
-        def initialize(id)
+        def initialize(id:)
           @id = id
-          super "Possible graph cycle discovered related to PID=#{id}."
+          super to_s
+        end
+
+        def to_s
+          "Possible graph cycle discovered related to ID=#{id.inspect}."
+        end
+      end
+
+      # Raised when we have exceeded the time to live constraint
+      # @see Samvera::NestingIndexer::Configuration.maximum_nesting_depth
+      class ExceededMaximumNestingDepthError < CycleDetectionError
+        def to_s
+          "Exceeded maximum nesting depth while indexing ID=#{id.inspect}."
+        end
+      end
+
+      # Raised when we encounter a document that is to be indexed as its own ancestor.
+      class DocumentIsItsOwnAncestorError < CycleDetectionError
+        attr_reader :pathnames
+        def initialize(id:, pathnames:)
+          super(id: id)
+          @pathnames = pathnames
+        end
+
+        def to_s
+          "Document with ID=#{id.inspect} is marked as its own ancestor based on the given pathnames: #{pathnames.inspect}."
         end
       end
       # A wrapper exception that includes the original exception and the id
@@ -41,7 +66,7 @@ module Samvera
         def initialize(id, original_exception)
           @id = id
           @original_exception = original_exception
-          super "Error PID=#{id} - #{original_exception}"
+          super "ReindexingError on ID=#{id.inspect}\n\t#{original_exception}"
         end
       end
     end

data/lib/samvera/nesting_indexer/relationship_reindexer.rb

@@ -6,7 +6,7 @@ module Samvera
   # Establishing namespace
   module NestingIndexer
     # Responsible for reindexing the PID and its descendants
-    # @note There is cycle detection via the TIME_TO_LIVE counter
+    # @note There is cycle detection via the Samvera::NestingIndexer::Configuration#maximum_nesting_depth counter
     # @api private
     class RelationshipReindexer
       # @api private
@@ -20,32 +20,38 @@ module Samvera
       end
 
       # @param id [String]
-      # @param maximum_nesting_depth [Integer] Samvera::NestingIndexer::TIME_TO_LIVE to detect cycles in the graph
-      # @param adapter [Samvera::NestingIndexer::Adapters::AbstractAdapter] Conforms to the Samvera::NestingIndexer::Adapters::AbstractAdapter interface
+      # @param maximum_nesting_depth [Integer] What is the maximum allowed depth of nesting
+      # @param configuration [#adapter, #logger] The :adapter conforms to the Samvera::NestingIndexer::Adapters::AbstractAdapter interface
+      #                                          and the :logger conforms to Logger
       # @param queue [#shift, #push] queue
-      def initialize(id:, maximum_nesting_depth:, adapter:, queue: [])
+      def initialize(id:, maximum_nesting_depth:, configuration:, queue: [], visited_ids: [])
         @id = id.to_s
         @maximum_nesting_depth = maximum_nesting_depth.to_i
-        @adapter = adapter
+        @configuration = configuration
         @queue = queue
+        @visited_ids = visited_ids
       end
-      attr_reader :id, :maximum_nesting_depth, :queue, :adapter
+      attr_reader :id, :maximum_nesting_depth
 
       # Perform a bread-first tree traversal of the initial document and its descendants.
+      # rubocop:disable Metrics/AbcSize
       def call
-        enqueue(initial_index_document, maximum_nesting_depth)
-        processing_document = dequeue
-        while processing_document
-          process_a_document(processing_document)
-          adapter.each_child_document_of(document: processing_document) { |child| enqueue(child, processing_document.maximum_nesting_depth - 1) }
+        wrap_logging("nested indexing of ID=#{initial_index_document.id.inspect}") do
+          enqueue(initial_index_document, maximum_nesting_depth)
           processing_document = dequeue
+          while processing_document
+            process_a_document(processing_document)
+            adapter.each_child_document_of(document: processing_document) { |child| enqueue(child, processing_document.maximum_nesting_depth - 1) }
+            processing_document = dequeue
+          end
         end
         self
       end
+      # rubocop:enbable Metrics/AbcSize
 
       private
 
-      attr_writer :document
+      attr_reader :queue, :configuration, :visited_ids
 
       def initial_index_document
         adapter.find_index_document_by(id: id)
@@ -53,6 +59,8 @@ module Samvera
 
       extend Forwardable
       def_delegator :queue, :shift, :dequeue
+      def_delegator :configuration, :adapter
+      def_delegator :configuration, :logger
 
       require 'delegate'
       # A small object to help track time to live concerns
@@ -69,16 +77,33 @@ module Samvera
       end
 
       def process_a_document(index_document)
-        raise Exceptions::CycleDetectionError, id if index_document.maximum_nesting_depth <= 0
-        preservation_document = adapter.find_preservation_document_by(id: index_document.id)
-        parent_ids_and_path_and_ancestors = parent_ids_and_path_and_ancestors_for(preservation_document)
-        adapter.write_document_attributes_to_index_layer(**parent_ids_and_path_and_ancestors)
+        raise Exceptions::ExceededMaximumNestingDepthError, id: id if index_document.maximum_nesting_depth <= 0
+        wrap_logging("indexing ID=#{index_document.id.inspect}") do
+          preservation_document = adapter.find_preservation_document_by(id: index_document.id)
+          parent_ids_and_path_and_ancestors = parent_ids_and_path_and_ancestors_for(preservation_document)
+          guard_against_possiblity_of_self_ancestry(index_document: index_document, pathnames: parent_ids_and_path_and_ancestors.fetch(:pathnames))
+          adapter.write_document_attributes_to_index_layer(**parent_ids_and_path_and_ancestors)
+          visited_ids << index_document.id
+        end
       end
 
       def parent_ids_and_path_and_ancestors_for(preservation_document)
         ParentAndPathAndAncestorsBuilder.new(preservation_document, adapter).to_hash
       end
 
+      def guard_against_possiblity_of_self_ancestry(index_document:, pathnames:)
+        pathnames.each do |pathname|
+          next unless pathname.include?("#{index_document.id}/")
+          raise Exceptions::DocumentIsItsOwnAncestorError, id: index_document.id, pathnames: pathnames
+        end
+      end
+
+      def wrap_logging(message_suffix)
+        logger.debug("Starting #{message_suffix}")
+        yield
+        logger.debug("Ending #{message_suffix}")
+      end
+
       # A small object that helps encapsulate the logic of building the hash of information regarding
       # the initialization of an Samvera::NestingIndexer::Documents::IndexDocument
       #
@@ -113,9 +138,9 @@ module Samvera
         def compile_one!(parent_index_document)
           @parent_ids << parent_index_document.id
           parent_index_document.pathnames.each do |pathname|
-            @pathnames << File.join(pathname, @preservation_document.id)
-            slugs = pathname.split("/")
-            slugs.each_index { |i| @ancestors << slugs[0..i].join('/') }
+            @pathnames << "#{pathname}#{Documents::ANCESTOR_AND_PATHNAME_DELIMITER}#{@preservation_document.id}"
+            slugs = pathname.split(Documents::ANCESTOR_AND_PATHNAME_DELIMITER)
+            slugs.each_index { |i| @ancestors << slugs[0..i].join(Documents::ANCESTOR_AND_PATHNAME_DELIMITER) }
           end
           @ancestors += parent_index_document.ancestors
         end

data/lib/samvera/nesting_indexer/repository_reindexer.rb

@@ -1,3 +1,5 @@
+require 'samvera/nesting_indexer/exceptions'
+require 'forwardable'
 module Samvera
   # Establishing namespace
   module NestingIndexer
@@ -19,24 +21,29 @@ module Samvera
 
       # @param id_reindexer [#call] Samvera::NestingIndexer.method(:reindex_relationships) Responsible for reindexing a single object
       # @param maximum_nesting_depth [Integer] detect cycles in the graph
-      # @param adapter [Samvera::NestingIndexer::Adapters::AbstractAdapter] Conforms to the Samvera::NestingIndexer::Adapters::AbstractAdapter interface
-      def initialize(maximum_nesting_depth:, id_reindexer:, adapter:)
+      # @param configuration [#adapter, #logger] The :adapter conforms to the Samvera::NestingIndexer::Adapters::AbstractAdapter interface
+      #                                          and the :logger conforms to Logger
+      def initialize(maximum_nesting_depth:, id_reindexer:, configuration:)
         @maximum_nesting_depth = maximum_nesting_depth.to_i
         @id_reindexer = id_reindexer
-        @adapter = adapter
+        @configuration = configuration
         @processed_ids = []
       end
 
       # @todo Would it make sense to leverage an each_preservation_id instead?
       def call
-        @adapter.each_perservation_document_id_and_parent_ids do |id, parent_ids|
+        adapter.each_perservation_document_id_and_parent_ids do |id, parent_ids|
           recursive_reindex(id: id, parent_ids: parent_ids, time_to_live: maximum_nesting_depth)
         end
       end
 
       private
 
-      attr_reader :maximum_nesting_depth, :processed_ids, :id_reindexer
+      attr_reader :maximum_nesting_depth, :processed_ids, :id_reindexer, :configuration
+
+      extend Forwardable
+      def_delegator :configuration, :adapter
+      def_delegator :configuration, :logger
 
       # When we find a document, reindex it if it doesn't have a parent. If it has a parent, reindex the parent first.
       #
@@ -47,9 +54,9 @@ module Samvera
       # walk up the parent graph to reindex the parents before we start on the child.
       def recursive_reindex(id:, parent_ids:, time_to_live:)
         return true if processed_ids.include?(id)
-        raise Exceptions::CycleDetectionError, id if time_to_live <= 0
+        raise Exceptions::ExceededMaximumNestingDepthError, id: id if time_to_live <= 0
         parent_ids.each do |parent_id|
-          grand_parent_ids = @adapter.find_preservation_parent_ids_for(id: parent_id)
+          grand_parent_ids = adapter.find_preservation_parent_ids_for(id: parent_id)
           recursive_reindex(id: parent_id, parent_ids: grand_parent_ids, time_to_live: maximum_nesting_depth - 1)
         end
         reindex_an_id(id)
@@ -59,6 +66,7 @@ module Samvera
         id_reindexer.call(id: id)
         processed_ids << id
       rescue StandardError => e
+        logger.error(e)
         raise Exceptions::ReindexingError.new(id, e)
       end
     end

data/lib/samvera/nesting_indexer/version.rb

@@ -1,5 +1,5 @@
 module Samvera
   module NestingIndexer
-    VERSION = "0.7.0".freeze
+    VERSION = "0.8.0".freeze
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: samvera-nesting_indexer
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.8.0
 platform: ruby
 authors:
 - Jeremy Friesen
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-08-25 00:00:00.000000000 Z
+date: 2017-11-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler