samvera-nesting_indexer 0.8.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 37f64367d992a4d794752c4eda559a3e066933f3
-  data.tar.gz: 5de4fc1861c009b962fecd391582bc2240f329b7
+  metadata.gz: bfb903c39e5837da3ce7162d2f16066b23b92127
+  data.tar.gz: 250a0310e9b7564e37e2e1ff7ac33b438fc83461
 SHA512:
-  metadata.gz: dad5de0e38c405e321e7f520cb0c44137423fa942e81218ba809e4dc9dc12255b15d69edc59f1c2b8aaa866504da064c1df1d45cd059fcf3e5069f234b2d2990
-  data.tar.gz: 6882d1ea6b0a40a9c738a420efe54ec3a984c591d16d20247bd3135f5a919883f249e13a3965585a7be9a6793e0ee5d1e83b656b8a858f832b71f7f14e195f6c
+  metadata.gz: 884293f1ed28ee9df34329d6799cab60fb7c356a914d04221814aa88b5e83ebd89a6f76d6076a5278c4547c93ecb88b9ddc50a52a14930c0e253d8be46ffa943
+  data.tar.gz: a53e33bec810987ea4d286d4e626e44aa6be24d72271a1c83994c149d47269c7751443e32ea80c62e2a281a3791f55ebc38052fbba409df34dc8a991bc7ead5c

data/README.md

@@ -34,7 +34,7 @@ We have four attributes to consider for indexing the graph:
 1. id - the unique identifier for a document
 2. parent_ids - the ids for all of the parents of a given document
 3. pathnames - the paths to traverse from a root document to the given document
-4. ancestors - the pathnames of each of the ancestors
+4. ancestors - the pathnames to each node that is an ancestor of the given node (e.g. pathname to my parent, pathname to my grandparent)
 
 See [Samvera::NestingIndexer::Documents::IndexDocument](./lib/samvera/nesting_indexer/documents.rb) for further discussion.
 
@@ -42,6 +42,8 @@ To reindex a single document, we leverage the [`Samvera::NestingIndexer.reindex_
 
 To reindex all of the documents, we leverage the [`Samvera::NestingIndexer.reindex_all!`](lib/samvera/nesting_indexer.rb) method. **Warning: This is a very slow process.**
 
+With a node's pathname(s), we are able to query what are all of my descendants (both direct and indirect) by way of the ancestors.
+
 ## Examples
 
 Given the following PreservationDocuments:
@@ -53,19 +55,24 @@ Given the following PreservationDocuments:
 | C   | A       |
 | D   | A, B    |
 | E   | C       |
+| F   | D       |
 
 If we were to reindex the above PreservationDocuments, we will generate the following IndexDocuments:
 
-| PID | Parents | Pathnames  | Ancestors |
-|-----|---------|------------|-----------|
-| A   | -       | [A]        | []        |
-| B   | -       | [B]        | []        |
-| C   | A       | [A/C]      | [A]       |
-| D   | A, B    | [A/D, B/D] | [A, B]    |
-| E   | C       | [A/C/E]    | [A/C]     |
+| PID | Parents | Pathnames      | Ancestors        |
+|-----|---------|----------------|------------------|
+| A   | -       | [A]            | []               |
+| B   | -       | [B]            | []               |
+| C   | A       | [A/C]          | [A]              |
+| D   | A, B    | [A/D, B/D]     | [A, B]           |
+| E   | C       | [A/C/E]        | [A, A/C]         |
+| F   | D       | [A/D/F, B/D/F] | [A, A/D, B, B/D] |
 
 For more scenarios, look at the [Reindex PID and Descendants specs](./spec/features/reindex_id_and_descendants_spec.rb).
 
+* Given I want to find the direct descendants of A, then I can query all nodes with that have a parent of A.
+* Given I want to find the direct and indirect descendants of A, then I can query all notes that have an ancestor entry of A.
+
 ## Adapters
 
 An [AbstractAdapter](./lib/samvera/nesting_indexer/adapters/abstract_adapter.rb) provides the method interface for others to build against.
@@ -95,6 +102,14 @@ end
 
 [See CurateND for Notre Dame's adaptor configuration](https://github.com/ndlib/samvera_nd/blob/6fbe79c9725c0f8b4641981044ec250c5163053b/config/initializers/samvera_config.rb#L32-L35).
 
+### Sequence Diagram for Reindexing a Single Document
+
+The following sequence diagram documents the interactions in [Samvera::NestingIndexer::RelationshipReindexer](lib/samvera/nesting_indexer/relationship_reindexer.rb).
+
+![Reindex Relationship Diagram](documentation/reindex_relationship.mermaid.jpg)
+
+See [the text-based version of Reindex Relationship diagram](documentation/reindex_relationship.mermaid), leveraging the [Mermaid syntax](https://mermaidjs.github.io).
+
 ## Considerations
 
 Given a single object A, when we reindex A, we:
@@ -115,9 +130,23 @@ The [`./spec/features/reindex_pid_and_descendants_spec.rb`](spec/features/reinde
 
 **NOTE: These guards to prevent indexing cyclic graphs do not prevent the underlying preservation document from creating its own cyclic graph.**
 
+#### Detecting Possible Cycles Before Indexing
+
+Given an up to date index and a document, then it is valid to nest the given document beneath any document that:
+
+* Is not the given document
+* Does not have one or more pathnames that includes the given document's ID
+
+For examples of determining if we can nest a document within another document, see the [demonstration of nesting](./spec/features/demonstrating_nesting_spec.rb).
+
+In implementations, you'll likely want to write a queries that answer:
+
+* What are the valid IDs that I can nest within?
+* What are the valid IDs in which I can nest within and am not already nested within?
+
 ## TODO
 
-- [ ] Incorporate additional logging
+- [X] Incorporate additional logging
 - [ ] Build methods to allow for fanning out the reindexing. At present, when we reindex a node and its "children", we run that entire process within a single context. Likewise, we run a single process when reindexing EVERYTHING.
 - [ ] Promote from [samvera-labs](https://github.com/samvera-labs) to [samvera](https://github.com/samvera) via the [promotion process](http://samvera-labs.github.io/promotion.html).
 - [ ] Write adapter method to assist in guarding against self-ancestry. We could probably expose a base adapter that has the method through use of the other adapter methods.

data/Rakefile

@@ -16,9 +16,7 @@ namespace :commitment do
     $stdout.puts "Checking commitment:code_coverage"
     coverage_percentage = JSON.parse(File.read('coverage/.last_run.json')).fetch('result').fetch('covered_percent').to_i
     goal = 100
-    if goal > coverage_percentage
-      abort("Code Coverage Goal Not Met:\n\t#{coverage_percentage}%\tExpected\n\t#{goal}%\tActual")
-    end
+    abort("Code Coverage Goal Not Met:\n\t#{coverage_percentage}%\tExpected\n\t#{goal}%\tActual") if goal > coverage_percentage
   end
 end
 

data/documentation/reindex_relationship.mermaid

@@ -0,0 +1,22 @@
+sequenceDiagram
+    participant Application
+    participant Indexer as Samvera:: NestingIndexer
+    participant Adapter as Application Adapter Implementation
+    participant Index as Application Index Layer
+    participant Preservation as Application Preservation Layer
+
+    Application-->>Indexer: indexer.reindex_relationships(id:)
+    Indexer-->>Adapter: adapter.find_index_document_by(id:)
+    Adapter-->>Index: get index document
+    Note right of Adapter: See Samvera:: NestingIndexer:: Adapters:: AbstractAdapter for adapter implementation
+    Index-->>Indexer: coerce index document to indexer
+    Indexer-->>Indexer: enqueue index document
+    loop While queued documents
+        Indexer-->>Adapter: adapter.find_preservation_document_by(id:)
+        Adapter-->>Preservation: get preservation document
+        Preservation-->>Indexer: receive preservation document
+        Indexer-->>Indexer: guard_against_possiblity_of_self_ancestry
+        Indexer-->>Adapter: adapter.write_document_attributes_to_index_layer
+        Adapter-->>Index: write updated application index document
+        Indexer-->>Indexer: enqueue children
+    end

data/lib/samvera/nesting_indexer/adapters/abstract_adapter.rb

@@ -43,13 +43,24 @@ module Samvera
         end
 
         # @api public
+        # @deprecated Use .write_nesting_document_to_index_layer instead
         # @see README.md
         # @param id [String]
         # @param parent_ids [Array<String>]
         # @param ancestors [Array<String>]
         # @param pathnames [Array<String>]
+        # @param deepest_nested_depth [Integer]
         # @return Hash - the attributes written to the indexing layer
-        def self.write_document_attributes_to_index_layer(id:, parent_ids:, ancestors:, pathnames:)
+        def self.write_document_attributes_to_index_layer(id:, parent_ids:, ancestors:, pathnames:, deepest_nested_depth:)
+          raise NotImplementedError
+        end
+
+        # @api public
+        # @since v1.0.0
+        # @see README.md
+        # @param nesting_document [Samvera::NestingIndexer::Document::IndexDocument]
+        # @return void
+        def self.write_nesting_document_to_index_layer(nesting_document:)
           raise NotImplementedError
         end
       end

data/lib/samvera/nesting_indexer/adapters/in_memory_adapter.rb

@@ -59,9 +59,17 @@ module Samvera
         # @param parent_ids [Array<String>]
         # @param ancestors [Array<String>]
         # @param pathnames [Array<String>]
+        # @param deepest_nested_depth [Integer]
         # @return [Hash] - the attributes written to the indexing layer
-        def self.write_document_attributes_to_index_layer(id:, parent_ids:, ancestors:, pathnames:)
-          Index.write_document(id: id, parent_ids: parent_ids, ancestors: ancestors, pathnames: pathnames)
+        def self.write_document_attributes_to_index_layer(id:, parent_ids:, ancestors:, pathnames:, deepest_nested_depth:)
+          Index.write_document(id: id, parent_ids: parent_ids, ancestors: ancestors, pathnames: pathnames, deepest_nested_depth: deepest_nested_depth)
+        end
+
+        # @api public
+        # @see README.md
+        # @param nesting_document [Samvera::NestingIndexer::Documents::IndexDocument]
+        def self.write_nesting_document_to_index_layer(nesting_document:)
+          Index.write_to_storage(nesting_document)
         end
 
         # @api private
@@ -70,6 +78,12 @@ module Samvera
           Index.clear_cache!
         end
 
+        # @api private
+        # A convenience method for testing
+        def self.each_index_document(&block)
+          Index.each_index_document(&block)
+        end
+
         # @api private
         #
         # A module mixin to expose rudimentary read/write capabilities
@@ -146,12 +160,20 @@ module Samvera
             Storage.find(id)
           end
 
+          def self.each_index_document(&block)
+            Storage.find_each(&block)
+          end
+
           def self.each_child_document_of(document:, &block)
             Storage.find_children_of_id(document.id).each(&block)
           end
 
+          def self.write_to_storage(doc)
+            Storage.write(doc)
+          end
+
           def self.write_document(attributes = {})
-            Documents::IndexDocument.new(attributes).tap { |doc| Storage.write(doc) }
+            Documents::IndexDocument.new(attributes).tap { |doc| write_to_storage(doc) }
           end
 
           # :nodoc:

data/lib/samvera/nesting_indexer/adapters/interface_behavior_spec.rb

@@ -82,8 +82,8 @@ if defined?(RSpec)
     describe '.write_document_attributes_to_index_layer' do
       subject { described_class.method(:write_document_attributes_to_index_layer) }
 
-      it 'requires the :ancestors, :id, :parent_ids, and :pathnames keyword (and does not require any others)' do
-        expect(required_keyword_parameters.call(subject)).to eq(%i(ancestors id parent_ids pathnames))
+      it 'requires the :ancestors, :deepest_nested_depth, :id, :parent_ids, and :pathnames keyword (and does not require any others)' do
+        expect(required_keyword_parameters.call(subject)).to eq(%i(ancestors deepest_nested_depth id parent_ids pathnames))
       end
 
       it 'does not require any other parameters (besides :attributes)' do
@@ -94,5 +94,21 @@ if defined?(RSpec)
         expect(block_parameter_extracter.call(subject)).to be_empty
       end
     end
+
+    describe '.write_nesting_document_to_index_layer' do
+      subject { described_class.method(:write_nesting_document_to_index_layer) }
+
+      it 'requires the :nesting_document' do
+        expect(required_keyword_parameters.call(subject)).to eq(%i(nesting_document))
+      end
+
+      it 'does not require any other parameters' do
+        expect(required_parameters.call(subject)).to eq(required_keyword_parameters.call(subject))
+      end
+
+      it 'does not expect a block' do
+        expect(block_parameter_extracter.call(subject)).to be_empty
+      end
+    end
   end
 end

data/lib/samvera/nesting_indexer/documents.rb

@@ -42,6 +42,19 @@ module Samvera
           @ancestors = Array(keywords.fetch(:ancestors))
         end
 
+        # @api public
+        # @since v1.0.0
+        # @return [Hash<Symbol,>] the Ruby hash representation of this index document.
+        def to_hash
+          {
+            id: id,
+            parent_ids: parent_ids,
+            pathnames: pathnames,
+            ancestors: ancestors,
+            deepest_nested_depth: deepest_nested_depth
+          }
+        end
+
         # @api public
         # @return String The Fedora object's PID
         attr_reader :id
@@ -70,11 +83,22 @@ module Samvera
         #
         # All of the :pathnames of each of the documents ancestors. If I have A, with parent B, and B has
         # parents C and D then we have the following ancestors:
-        #   [D/B], [C/B]
+        #   [D], [C], [D/B], [C/B]
         #
         # @return Array<String>
         attr_reader :ancestors
 
+        # @api public
+        # @since v1.0.0
+        #
+        # The largest nesting depth of this document. If I have A ={ B ={ C and D ={ C, then
+        # deepest_nested_depth is 3.
+        #
+        # @return Integer
+        def deepest_nested_depth
+          pathnames.map(&:length).max
+        end
+
         def sorted_parent_ids
           parent_ids.sort
         end

data/lib/samvera/nesting_indexer/relationship_reindexer.rb

@@ -5,7 +5,7 @@ require 'set'
 module Samvera
   # Establishing namespace
   module NestingIndexer
-    # Responsible for reindexing the PID and its descendants
+    # Responsible for reindexing the document associated with the given PID and its descendant documents
     # @note There is cycle detection via the Samvera::NestingIndexer::Configuration#maximum_nesting_depth counter
     # @api private
     class RelationshipReindexer
@@ -24,34 +24,38 @@ module Samvera
       # @param configuration [#adapter, #logger] The :adapter conforms to the Samvera::NestingIndexer::Adapters::AbstractAdapter interface
       #                                          and the :logger conforms to Logger
       # @param queue [#shift, #push] queue
-      def initialize(id:, maximum_nesting_depth:, configuration:, queue: [], visited_ids: [])
+      def initialize(id:, maximum_nesting_depth:, configuration:, queue: [])
         @id = id.to_s
         @maximum_nesting_depth = maximum_nesting_depth.to_i
         @configuration = configuration
         @queue = queue
-        @visited_ids = visited_ids
       end
       attr_reader :id, :maximum_nesting_depth
 
-      # Perform a bread-first tree traversal of the initial document and its descendants.
-      # rubocop:disable Metrics/AbcSize
+      # Perform a breadth-first tree traversal of the initial document and its descendants.
+      # We index the document, then queue up each of its children. For each child, queue up the child's children.
       def call
         wrap_logging("nested indexing of ID=#{initial_index_document.id.inspect}") do
           enqueue(initial_index_document, maximum_nesting_depth)
-          processing_document = dequeue
-          while processing_document
-            process_a_document(processing_document)
-            adapter.each_child_document_of(document: processing_document) { |child| enqueue(child, processing_document.maximum_nesting_depth - 1) }
-            processing_document = dequeue
-          end
+          process_each_document
         end
         self
       end
-      # rubocop:enbable Metrics/AbcSize
 
       private
 
-      attr_reader :queue, :configuration, :visited_ids
+      attr_reader :queue, :configuration
+
+      def process_each_document
+        processing_document = dequeue
+        while processing_document
+          process_a_document(processing_document)
+          adapter.each_child_document_of(document: processing_document) do |child|
+            enqueue(child, processing_document.maximum_nesting_depth - 1)
+          end
+          processing_document = dequeue
+        end
+      end
 
       def initial_index_document
         adapter.find_index_document_by(id: id)
@@ -76,19 +80,20 @@ module Samvera
         queue.push(ProcessingDocument.new(document, maximum_nesting_depth))
       end
 
+      # rubocop:disable Metrics/AbcSize
       def process_a_document(index_document)
         raise Exceptions::ExceededMaximumNestingDepthError, id: id if index_document.maximum_nesting_depth <= 0
         wrap_logging("indexing ID=#{index_document.id.inspect}") do
           preservation_document = adapter.find_preservation_document_by(id: index_document.id)
-          parent_ids_and_path_and_ancestors = parent_ids_and_path_and_ancestors_for(preservation_document)
-          guard_against_possiblity_of_self_ancestry(index_document: index_document, pathnames: parent_ids_and_path_and_ancestors.fetch(:pathnames))
-          adapter.write_document_attributes_to_index_layer(**parent_ids_and_path_and_ancestors)
-          visited_ids << index_document.id
+          nesting_document = build_nesting_document_for(preservation_document)
+          guard_against_possiblity_of_self_ancestry(index_document: index_document, pathnames: nesting_document.pathnames)
+          adapter.write_nesting_document_to_index_layer(nesting_document: nesting_document)
         end
       end
+      # rubocop:enable Metrics/AbcSize
 
-      def parent_ids_and_path_and_ancestors_for(preservation_document)
-        ParentAndPathAndAncestorsBuilder.new(preservation_document, adapter).to_hash
+      def build_nesting_document_for(preservation_document)
+        ParentAndPathAndAncestorsBuilder.new(preservation_document, adapter).nesting_document
       end
 
       def guard_against_possiblity_of_self_ancestry(index_document:, pathnames:)
@@ -104,7 +109,7 @@ module Samvera
         logger.debug("Ending #{message_suffix}")
       end
 
-      # A small object that helps encapsulate the logic of building the hash of information regarding
+      # A small object that helps encapsulate the logic for building the hash of information regarding
       # the initialization of an Samvera::NestingIndexer::Documents::IndexDocument
       #
       # @see Samvera::NestingIndexer::Documents::IndexDocument for details on pathnames, ancestors, and parent_ids.
@@ -116,11 +121,10 @@ module Samvera
           @ancestors = Set.new
           @adapter = adapter
           compile!
+          @nesting_document = Documents::IndexDocument.new(id: @preservation_document.id, parent_ids: @parent_ids, pathnames: @pathnames, ancestors: @ancestors)
         end
 
-        def to_hash
-          { id: @preservation_document.id, parent_ids: @parent_ids.to_a, pathnames: @pathnames.to_a, ancestors: @ancestors.to_a }
-        end
+        attr_reader :nesting_document
 
         private
 
@@ -140,7 +144,9 @@ module Samvera
           parent_index_document.pathnames.each do |pathname|
             @pathnames << "#{pathname}#{Documents::ANCESTOR_AND_PATHNAME_DELIMITER}#{@preservation_document.id}"
             slugs = pathname.split(Documents::ANCESTOR_AND_PATHNAME_DELIMITER)
-            slugs.each_index { |i| @ancestors << slugs[0..i].join(Documents::ANCESTOR_AND_PATHNAME_DELIMITER) }
+            slugs.each_index do |i|
+              @ancestors << slugs[0..i].join(Documents::ANCESTOR_AND_PATHNAME_DELIMITER)
+            end
           end
           @ancestors += parent_index_document.ancestors
         end

data/lib/samvera/nesting_indexer/version.rb

@@ -1,5 +1,5 @@
 module Samvera
   module NestingIndexer
-    VERSION = "0.8.0".freeze
+    VERSION = "1.0.0".freeze
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: samvera-nesting_indexer
 version: !ruby/object:Gem::Version
-  version: 0.8.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Jeremy Friesen
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-11-17 00:00:00.000000000 Z
+date: 2018-01-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -254,6 +254,8 @@ files:
 - Rakefile
 - bin/console
 - bin/setup
+- documentation/reindex_relationship.mermaid
+- documentation/reindex_relationship.mermaid.jpg
 - lib/samvera/nesting_indexer.rb
 - lib/samvera/nesting_indexer/adapters.rb
 - lib/samvera/nesting_indexer/adapters/abstract_adapter.rb