bulkrax 5.2.1 → 5.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +6 -4
- data/app/assets/javascripts/bulkrax/navtabs.js.erb +9 -0
- data/app/controllers/bulkrax/exporters_controller.rb +2 -1
- data/app/controllers/bulkrax/importers_controller.rb +2 -1
- data/app/jobs/bulkrax/create_relationships_job.rb +59 -16
- data/app/matchers/bulkrax/application_matcher.rb +4 -3
- data/app/models/bulkrax/csv_entry.rb +18 -13
- data/app/models/bulkrax/importer.rb +2 -3
- data/app/parsers/bulkrax/csv_parser.rb +2 -2
- data/app/parsers/bulkrax/parser_export_record_set.rb +72 -55
- data/app/views/bulkrax/importers/edit.html.erb +4 -2
- data/app/views/bulkrax/importers/new.html.erb +4 -2
- data/app/views/hyrax/dashboard/sidebar/_bulkrax_sidebar_additions.html.erb +4 -2
- data/app/views/hyrax/dashboard/sidebar/_repository_content.html.erb +4 -2
- data/db/migrate/20230608153601_add_indices_to_bulkrax.rb +14 -0
- data/lib/bulkrax/version.rb +1 -1
- data/lib/bulkrax.rb +13 -1
- data/lib/generators/bulkrax/templates/config/initializers/bulkrax.rb +2 -2
- metadata +47 -3
    
        checksums.yaml
    CHANGED
    
    | @@ -1,7 +1,7 @@ | |
| 1 1 | 
             
            ---
         | 
| 2 2 | 
             
            SHA256:
         | 
| 3 | 
            -
              metadata.gz:  | 
| 4 | 
            -
              data.tar.gz:  | 
| 3 | 
            +
              metadata.gz: 687e8437db3ceb118fd59fae8d1350379749ec42c60f236833b3b198376281c7
         | 
| 4 | 
            +
              data.tar.gz: 9f4af8b8e5eda6b48f8a97c7a34216a41476df44b723e54d623b3db6584edcf8
         | 
| 5 5 | 
             
            SHA512:
         | 
| 6 | 
            -
              metadata.gz:  | 
| 7 | 
            -
              data.tar.gz:  | 
| 6 | 
            +
              metadata.gz: 11c7518861a687b5db06ca319f5c3a0c0d9cc6621f55f17e9bb02bdd83f69fb18093f16235c5bcc06a50864b0acfd9f4c242d93477646ca782fdba62cf9ad1d8
         | 
| 7 | 
            +
              data.tar.gz: b3728de99a37f6846e83fd27c54be3c3fccfe03d256296f0a3123311797d7ab8eaa8be7a197de3657532adaadd58dd15fadd6b639158f43d1c77512f258cbfce
         | 
    
        data/README.md
    CHANGED
    
    | @@ -17,6 +17,7 @@ And then execute: | |
| 17 17 | 
             
            ```bash
         | 
| 18 18 | 
             
            $ bundle install
         | 
| 19 19 | 
             
            $ rails generate bulkrax:install
         | 
| 20 | 
            +
            $ rails db:migrate
         | 
| 20 21 | 
             
            ```
         | 
| 21 22 |  | 
| 22 23 | 
             
            If using Sidekiq, set up queues for `import` and `export`.
         | 
| @@ -32,6 +33,7 @@ gem 'bulkrax' | |
| 32 33 | 
             
            And then execute:
         | 
| 33 34 | 
             
            ```bash
         | 
| 34 35 | 
             
            $ bundle install
         | 
| 36 | 
            +
            $ rails db:migrate
         | 
| 35 37 | 
             
            ```
         | 
| 36 38 |  | 
| 37 39 | 
             
            Mount the engine in your routes file
         | 
| @@ -61,7 +63,7 @@ If using Sidekiq, set up queues for `import` and `export`. | |
| 61 63 | 
             
            *= require 'bulkrax/application'
         | 
| 62 64 | 
             
            ```
         | 
| 63 65 |  | 
| 64 | 
            -
            You'll want to add an  | 
| 66 | 
            +
            You'll want to add an initializer to configure the importer to your needs:
         | 
| 65 67 |  | 
| 66 68 | 
             
            ```ruby
         | 
| 67 69 | 
             
            # config/initializers/bulkrax.rb
         | 
| @@ -112,13 +114,13 @@ An Import needs to know what Work Type to create. The importer looks for: | |
| 112 114 |  | 
| 113 115 | 
             
            If it does not find either of these, or the data they contain is not a valid Work Type in the repository, the `default_work_type` will be used.
         | 
| 114 116 |  | 
| 115 | 
            -
            The install generator sets `default_work_type` to the first Work Type returned by `Hyrax.config.curation_concerns` but this can be  | 
| 117 | 
            +
            The install generator sets `default_work_type` to the first Work Type returned by `Hyrax.config.curation_concerns` (stringified), but this can be overwritten by setting `default_work_type` in `config/initializer/bulkrax.rb` as shown above.
         | 
| 116 118 |  | 
| 117 119 | 
             
            ## Configuring Field Mapping
         | 
| 118 120 |  | 
| 119 121 | 
             
            It's unlikely that the incoming import data has fields that exactly match those in your repository. Field mappings allow you to tell bulkrax how to map field in the incoming data to a field in your application.
         | 
| 120 122 |  | 
| 121 | 
            -
            By default, a mapping for the OAI parser has been added to map standard oai_dc fields to Hyrax basic_metadata. The other parsers have no default mapping, and will map any incoming fields to Hyrax properties with the same name. Configurations can be added in `config/ | 
| 123 | 
            +
            By default, a mapping for the OAI parser has been added to map standard oai_dc fields to Hyrax basic_metadata. The other parsers have no default mapping, and will map any incoming fields to Hyrax properties with the same name. Configurations can be added in `config/initializers/bulkrax.rb`
         | 
| 122 124 |  | 
| 123 125 | 
             
            Configuring field mappings is documented in the [Bulkrax Configuration Guide](https://github.com/samvera-labs/bulkrax/wiki/Configuring-Bulkrax).
         | 
| 124 126 |  | 
| @@ -176,7 +178,7 @@ To edit an importer or exporter, select the edit icon (pencil) and complete the | |
| 176 178 | 
             
            To delete an importer or exporter, select the delete (x) icon.
         | 
| 177 179 |  | 
| 178 180 | 
             
            ### Downloading an export
         | 
| 179 | 
            -
            Once your the exporter has run, a download icon will  | 
| 181 | 
            +
            Once your the exporter has run, a download icon will appear on the exporters menu page.
         | 
| 180 182 |  | 
| 181 183 | 
             
            ## Contributing
         | 
| 182 184 | 
             
            If you're working on a PR for this project, create a feature branch off of `main`.
         | 
| @@ -61,38 +61,82 @@ module Bulkrax | |
| 61 61 | 
             
                  number_of_successes = 0
         | 
| 62 62 | 
             
                  number_of_failures = 0
         | 
| 63 63 | 
             
                  errors = []
         | 
| 64 | 
            -
             | 
| 65 | 
            -
                   | 
| 66 | 
            -
             | 
| 67 | 
            -
             | 
| 68 | 
            -
             | 
| 69 | 
            -
                       | 
| 70 | 
            -
             | 
| 71 | 
            -
             | 
| 72 | 
            -
             | 
| 64 | 
            +
                  @parent_record_members_added = false
         | 
| 65 | 
            +
                  @child_members_added = []
         | 
| 66 | 
            +
             | 
| 67 | 
            +
                  if parent_record
         | 
| 68 | 
            +
                    conditionally_acquire_lock_for(parent_record.id) do
         | 
| 69 | 
            +
                      ActiveRecord::Base.uncached do
         | 
| 70 | 
            +
                        Bulkrax::PendingRelationship.where(parent_id: parent_identifier, importer_run_id: importer_run_id)
         | 
| 71 | 
            +
                                                    .ordered.find_each do |rel|
         | 
| 72 | 
            +
                          process(relationship: rel, importer_run_id: importer_run_id, parent_record: parent_record, ability: ability)
         | 
| 73 | 
            +
                          number_of_successes += 1
         | 
| 74 | 
            +
                        rescue => e
         | 
| 75 | 
            +
                          number_of_failures += 1
         | 
| 76 | 
            +
                          errors << e
         | 
| 77 | 
            +
                        end
         | 
| 78 | 
            +
                      end
         | 
| 79 | 
            +
             | 
| 80 | 
            +
                      # save record if members were added
         | 
| 81 | 
            +
                      if @parent_record_members_added
         | 
| 82 | 
            +
                        parent_record.save!
         | 
| 83 | 
            +
                        # Ensure that the new relationship gets indexed onto the children
         | 
| 84 | 
            +
                        @child_members_added.each(&:update_index)
         | 
| 85 | 
            +
                      end
         | 
| 73 86 | 
             
                    end
         | 
| 87 | 
            +
                  else
         | 
| 88 | 
            +
                    # In moving the check of the parent record "up" we've exposed a hidden reporting foible.
         | 
| 89 | 
            +
                    # Namely we were reporting one error per child record when the parent record was itself
         | 
| 90 | 
            +
                    # unavailable.
         | 
| 91 | 
            +
                    #
         | 
| 92 | 
            +
                    # We have chosen not to duplicate that "number of errors" as it does not seem like the
         | 
| 93 | 
            +
                    # correct pattern for reporting a singular error (the previous pattern being one error per
         | 
| 94 | 
            +
                    # child who's parent is not yet created).
         | 
| 95 | 
            +
                    number_of_failures = 1
         | 
| 96 | 
            +
                    errors = ["Parent record not yet available for creating relationships with children records."]
         | 
| 74 97 | 
             
                  end
         | 
| 75 98 |  | 
| 76 | 
            -
                  # save record if members were added
         | 
| 77 | 
            -
                  parent_record.save! if @parent_record_members_added
         | 
| 78 | 
            -
             | 
| 79 | 
            -
                  # rubocop:disable Rails/SkipsModelValidations
         | 
| 80 99 | 
             
                  if errors.present?
         | 
| 100 | 
            +
                    # rubocop:disable Rails/SkipsModelValidations
         | 
| 81 101 | 
             
                    importer_run.increment!(:failed_relationships, number_of_failures)
         | 
| 102 | 
            +
                    # rubocop:enable Rails/SkipsModelValidations
         | 
| 103 | 
            +
             | 
| 82 104 | 
             
                    parent_entry&.set_status_info(errors.last, importer_run)
         | 
| 83 105 |  | 
| 84 106 | 
             
                    # TODO: This can create an infinite job cycle, consider a time to live tracker.
         | 
| 85 107 | 
             
                    reschedule({ parent_identifier: parent_identifier, importer_run_id: importer_run_id })
         | 
| 86 108 | 
             
                    return false # stop current job from continuing to run after rescheduling
         | 
| 87 109 | 
             
                  else
         | 
| 110 | 
            +
                    # rubocop:disable Rails/SkipsModelValidations
         | 
| 88 111 | 
             
                    Bulkrax::ImporterRun.find(importer_run_id).increment!(:processed_relationships, number_of_successes)
         | 
| 112 | 
            +
                    # rubocop:enable Rails/SkipsModelValidations
         | 
| 89 113 | 
             
                  end
         | 
| 90 | 
            -
                  # rubocop:enable Rails/SkipsModelValidations
         | 
| 91 114 | 
             
                end
         | 
| 92 115 | 
             
                # rubocop:enable Metrics/MethodLength
         | 
| 93 116 |  | 
| 94 117 | 
             
                private
         | 
| 95 118 |  | 
| 119 | 
            +
                ##
         | 
| 120 | 
            +
                # We can use Hyrax's lock manager when we have one available.
         | 
| 121 | 
            +
                if defined?(::Hyrax)
         | 
| 122 | 
            +
                  include Hyrax::Lockable
         | 
| 123 | 
            +
             | 
| 124 | 
            +
                  def conditionally_acquire_lock_for(*args, &block)
         | 
| 125 | 
            +
                    if Bulkrax.use_locking?
         | 
| 126 | 
            +
                      acquire_lock_for(*args, &block)
         | 
| 127 | 
            +
                    else
         | 
| 128 | 
            +
                      yield
         | 
| 129 | 
            +
                    end
         | 
| 130 | 
            +
                  end
         | 
| 131 | 
            +
                else
         | 
| 132 | 
            +
                  # Otherwise, we're providing no meaningful lock manager at this time.
         | 
| 133 | 
            +
                  def acquire_lock_for(*)
         | 
| 134 | 
            +
                    yield
         | 
| 135 | 
            +
                  end
         | 
| 136 | 
            +
             | 
| 137 | 
            +
                  alias conditionally_acquire_lock_for acquire_lock_for
         | 
| 138 | 
            +
                end
         | 
| 139 | 
            +
             | 
| 96 140 | 
             
                def process(relationship:, importer_run_id:, parent_record:, ability:)
         | 
| 97 141 | 
             
                  raise "#{relationship} needs a child to create relationship" if relationship.child_id.nil?
         | 
| 98 142 | 
             
                  raise "#{relationship} needs a parent to create relationship" if relationship.parent_id.nil?
         | 
| @@ -124,8 +168,7 @@ module Bulkrax | |
| 124 168 |  | 
| 125 169 | 
             
                  parent_record.ordered_members << child_record
         | 
| 126 170 | 
             
                  @parent_record_members_added = true
         | 
| 127 | 
            -
                   | 
| 128 | 
            -
                  child_record.save!
         | 
| 171 | 
            +
                  @child_members_added << child_record
         | 
| 129 172 | 
             
                end
         | 
| 130 173 |  | 
| 131 174 | 
             
                def reschedule(parent_identifier:, importer_run_id:)
         | 
| @@ -26,7 +26,8 @@ module Bulkrax | |
| 26 26 |  | 
| 27 27 | 
             
                  # @result will evaluate to an empty string for nil content values
         | 
| 28 28 | 
             
                  @result = content.to_s.gsub(/\s/, ' ').strip # remove any line feeds and tabs
         | 
| 29 | 
            -
                   | 
| 29 | 
            +
                  # blank needs to be based to split, only skip nil
         | 
| 30 | 
            +
                  process_split unless @result.nil?
         | 
| 30 31 | 
             
                  @result = @result[0] if @result.is_a?(Array) && @result.size == 1
         | 
| 31 32 | 
             
                  process_parse
         | 
| 32 33 | 
             
                  return @result
         | 
| @@ -36,8 +37,8 @@ module Bulkrax | |
| 36 37 | 
             
                  if self.split.is_a?(TrueClass)
         | 
| 37 38 | 
             
                    @result = @result.split(Bulkrax.multi_value_element_split_on)
         | 
| 38 39 | 
             
                  elsif self.split
         | 
| 39 | 
            -
                    result = @result.split(Regexp.new(self.split))
         | 
| 40 | 
            -
                    @result = result.map(&:strip)
         | 
| 40 | 
            +
                    @result = @result.split(Regexp.new(self.split))
         | 
| 41 | 
            +
                    @result = @result.map(&:strip).select(&:present?)
         | 
| 41 42 | 
             
                  end
         | 
| 42 43 | 
             
                end
         | 
| 43 44 |  | 
| @@ -20,7 +20,7 @@ module Bulkrax | |
| 20 20 | 
             
                  raise StandardError, 'CSV path empty' if path.blank?
         | 
| 21 21 | 
             
                  options = {
         | 
| 22 22 | 
             
                    headers: true,
         | 
| 23 | 
            -
                    header_converters: ->(h) { h.to_sym },
         | 
| 23 | 
            +
                    header_converters: ->(h) { h.to_s.strip.to_sym },
         | 
| 24 24 | 
             
                    encoding: 'utf-8'
         | 
| 25 25 | 
             
                  }.merge(csv_read_data_options)
         | 
| 26 26 |  | 
| @@ -243,20 +243,17 @@ module Bulkrax | |
| 243 243 | 
             
                  object_metadata(Array.wrap(data))
         | 
| 244 244 | 
             
                end
         | 
| 245 245 |  | 
| 246 | 
            -
                def build_value( | 
| 247 | 
            -
                  return unless hyrax_record.respond_to?( | 
| 246 | 
            +
                def build_value(property_name, mapping_config)
         | 
| 247 | 
            +
                  return unless hyrax_record.respond_to?(property_name.to_s)
         | 
| 248 248 |  | 
| 249 | 
            -
                  data = hyrax_record.send( | 
| 250 | 
            -
             | 
| 251 | 
            -
             | 
| 252 | 
            -
             | 
| 253 | 
            -
                    else
         | 
| 254 | 
            -
                      data.each_with_index do |d, i|
         | 
| 255 | 
            -
                        self.parsed_metadata["#{key_for_export(key)}_#{i + 1}"] = prepare_export_data(d)
         | 
| 256 | 
            -
                      end
         | 
| 257 | 
            -
                    end
         | 
| 249 | 
            +
                  data = hyrax_record.send(property_name.to_s)
         | 
| 250 | 
            +
             | 
| 251 | 
            +
                  if mapping_config['join'] || !data.is_a?(Enumerable)
         | 
| 252 | 
            +
                    self.parsed_metadata[key_for_export(property_name)] = prepare_export_data_with_join(data)
         | 
| 258 253 | 
             
                  else
         | 
| 259 | 
            -
                     | 
| 254 | 
            +
                    data.each_with_index do |d, i|
         | 
| 255 | 
            +
                      self.parsed_metadata["#{key_for_export(property_name)}_#{i + 1}"] = prepare_export_data(d)
         | 
| 256 | 
            +
                    end
         | 
| 260 257 | 
             
                  end
         | 
| 261 258 | 
             
                end
         | 
| 262 259 |  | 
| @@ -269,6 +266,14 @@ module Bulkrax | |
| 269 266 | 
             
                  "#{unnumbered_key}#{key.sub(clean_key, '')}"
         | 
| 270 267 | 
             
                end
         | 
| 271 268 |  | 
| 269 | 
            +
                def prepare_export_data_with_join(data)
         | 
| 270 | 
            +
                  # Yes...it's possible we're asking to coerce a multi-value but only have a single value.
         | 
| 271 | 
            +
                  return data.to_s unless data.is_a?(Enumerable)
         | 
| 272 | 
            +
                  return "" if data.empty?
         | 
| 273 | 
            +
             | 
| 274 | 
            +
                  data.map { |d| prepare_export_data(d) }.join(Bulkrax.multi_value_element_join_on).to_s
         | 
| 275 | 
            +
                end
         | 
| 276 | 
            +
             | 
| 272 277 | 
             
                def prepare_export_data(datum)
         | 
| 273 278 | 
             
                  if datum.is_a?(ActiveTriples::Resource)
         | 
| 274 279 | 
             
                    datum.to_uri.to_s
         | 
| @@ -24,13 +24,12 @@ module Bulkrax | |
| 24 24 | 
             
                attr_writer :current_run
         | 
| 25 25 |  | 
| 26 26 | 
             
                def self.safe_uri_filename(uri)
         | 
| 27 | 
            -
                  uri = URI.parse(uri) unless uri.is_a?(URI)
         | 
| 28 27 | 
             
                  r = Faraday.head(uri.to_s)
         | 
| 29 28 | 
             
                  return CGI.parse(r.headers['content-disposition'])["filename"][0].delete("\"")
         | 
| 30 29 | 
             
                rescue
         | 
| 31 | 
            -
                  filename = File.basename(uri. | 
| 30 | 
            +
                  filename = File.basename(uri.to_s)
         | 
| 32 31 | 
             
                  filename.delete!('/')
         | 
| 33 | 
            -
                  filename.presence ||  | 
| 32 | 
            +
                  filename.presence || SecureRandom.uuid
         | 
| 34 33 | 
             
                end
         | 
| 35 34 |  | 
| 36 35 | 
             
                def status
         | 
| @@ -33,9 +33,9 @@ module Bulkrax | |
| 33 33 | 
             
                      model_field_mappings.map(&:to_sym).each do |model_mapping|
         | 
| 34 34 | 
             
                        next unless r.key?(model_mapping)
         | 
| 35 35 |  | 
| 36 | 
            -
                        if r[model_mapping].casecmp('collection').zero?
         | 
| 36 | 
            +
                        if r[model_mapping].strip.casecmp('collection').zero?
         | 
| 37 37 | 
             
                          @collections << r
         | 
| 38 | 
            -
                        elsif r[model_mapping].casecmp('fileset').zero?
         | 
| 38 | 
            +
                        elsif r[model_mapping].strip.casecmp('fileset').zero?
         | 
| 39 39 | 
             
                          @file_sets << r
         | 
| 40 40 | 
             
                        else
         | 
| 41 41 | 
             
                          @works << r
         | 
| @@ -21,6 +21,30 @@ module Bulkrax | |
| 21 21 | 
             
                  "Bulkrax::ParserExportRecordSet::#{export_from.classify}".constantize.new(parser: parser)
         | 
| 22 22 | 
             
                end
         | 
| 23 23 |  | 
| 24 | 
            +
                SOLR_QUERY_PAGE_SIZE = 512
         | 
| 25 | 
            +
             | 
| 26 | 
            +
                ##
         | 
| 27 | 
            +
                # A helper method for handling querying large batches of IDs.  By default SOLR has a max of 1024
         | 
| 28 | 
            +
                # `OR` clauses per query.  This method helps chunk large sets of IDs into batches.
         | 
| 29 | 
            +
                #
         | 
| 30 | 
            +
                # @param array [Array<Object>]
         | 
| 31 | 
            +
                # @param page_size [Integer]
         | 
| 32 | 
            +
                # @yieldparam [Array<Object>] slice of the original arrays which are yielded.  The results of
         | 
| 33 | 
            +
                #             the yield are merged into the return value.
         | 
| 34 | 
            +
                #
         | 
| 35 | 
            +
                # @return [Array<Object>]
         | 
| 36 | 
            +
                #
         | 
| 37 | 
            +
                # @see https://github.com/samvera-labs/bulkrax/issues/776
         | 
| 38 | 
            +
                def self.in_batches(array, page_size: SOLR_QUERY_PAGE_SIZE)
         | 
| 39 | 
            +
                  array = Array.wrap(array)
         | 
| 40 | 
            +
                  return [] if array.empty?
         | 
| 41 | 
            +
                  results = []
         | 
| 42 | 
            +
                  array.each_slice(page_size) do |slice|
         | 
| 43 | 
            +
                    results += Array.wrap(yield(slice))
         | 
| 44 | 
            +
                  end
         | 
| 45 | 
            +
                  results
         | 
| 46 | 
            +
                end
         | 
| 47 | 
            +
             | 
| 24 48 | 
             
                # @abstract
         | 
| 25 49 | 
             
                #
         | 
| 26 50 | 
             
                # @note This has {#each} and {#count} but is not an Enumerable.  But because it has these two
         | 
| @@ -36,6 +60,7 @@ module Bulkrax | |
| 36 60 | 
             
                  delegate :limit_reached?, :work_entry_class, :collection_entry_class, :file_set_entry_class, :importerexporter, to: :parser
         | 
| 37 61 | 
             
                  private :limit_reached?, :work_entry_class, :collection_entry_class, :file_set_entry_class, :importerexporter
         | 
| 38 62 |  | 
| 63 | 
            +
                  ##
         | 
| 39 64 | 
             
                  # @return [Integer]
         | 
| 40 65 | 
             
                  def count
         | 
| 41 66 | 
             
                    sum = works.count + collections.count + file_sets.count
         | 
| @@ -44,6 +69,7 @@ module Bulkrax | |
| 44 69 | 
             
                    return sum
         | 
| 45 70 | 
             
                  end
         | 
| 46 71 |  | 
| 72 | 
            +
                  ##
         | 
| 47 73 | 
             
                  # Yield first the works, then collections, then file sets.  Once we've yielded as many times
         | 
| 48 74 | 
             
                  # as the parser's limit, we break the iteration and return.
         | 
| 49 75 | 
             
                  #
         | 
| @@ -134,8 +160,6 @@ module Bulkrax | |
| 134 160 | 
             
                                     end
         | 
| 135 161 | 
             
                  end
         | 
| 136 162 |  | 
| 137 | 
            -
                  SOLR_QUERY_PAGE_SIZE = 512
         | 
| 138 | 
            -
             | 
| 139 163 | 
             
                  # @note In most cases, when we don't have any candidate file sets, there is no need to query SOLR.
         | 
| 140 164 | 
             
                  #
         | 
| 141 165 | 
             
                  # @see Bulkrax::ParserExportRecordSet::Importer#file_sets
         | 
| @@ -148,20 +172,14 @@ module Bulkrax | |
| 148 172 | 
             
                  # @see https://github.com/scientist-softserv/britishlibrary/issues/289
         | 
| 149 173 | 
             
                  # @see https://github.com/samvera/hyrax/blob/64c0bbf0dc0d3e1b49f040b50ea70d177cc9d8f6/app/indexers/hyrax/work_indexer.rb#L15-L18
         | 
| 150 174 | 
             
                  def file_sets
         | 
| 151 | 
            -
                    @file_sets ||=  | 
| 152 | 
            -
             | 
| 153 | 
            -
             | 
| 154 | 
            -
             | 
| 155 | 
            -
             | 
| 156 | 
            -
             | 
| 157 | 
            -
             | 
| 158 | 
            -
             | 
| 159 | 
            -
                                         fsq,
         | 
| 160 | 
            -
                                         { fl: "id", method: :post, rows: ids.size }
         | 
| 161 | 
            -
                                       )
         | 
| 162 | 
            -
                                     end
         | 
| 163 | 
            -
                                     results
         | 
| 164 | 
            -
                                   end
         | 
| 175 | 
            +
                    @file_sets ||= ParserExportRecordSet.in_batches(candidate_file_set_ids) do |batch_of_ids|
         | 
| 176 | 
            +
                      fsq = "has_model_ssim:#{Bulkrax.file_model_class} AND id:(\"" + batch_of_ids.join('" OR "') + "\")"
         | 
| 177 | 
            +
                      fsq += extra_filters if extra_filters.present?
         | 
| 178 | 
            +
                      ActiveFedora::SolrService.query(
         | 
| 179 | 
            +
                        fsq,
         | 
| 180 | 
            +
                        { fl: "id", method: :post, rows: batch_of_ids.size }
         | 
| 181 | 
            +
                      )
         | 
| 182 | 
            +
                    end
         | 
| 165 183 | 
             
                  end
         | 
| 166 184 |  | 
| 167 185 | 
             
                  def solr_name(base_name)
         | 
| @@ -227,32 +245,34 @@ module Bulkrax | |
| 227 245 | 
             
                      end
         | 
| 228 246 | 
             
                  end
         | 
| 229 247 |  | 
| 230 | 
            -
                  def  | 
| 231 | 
            -
                     | 
| 232 | 
            -
                       | 
| 233 | 
            -
                         | 
| 234 | 
            -
                         | 
| 235 | 
            -
             | 
| 236 | 
            -
             | 
| 237 | 
            -
             | 
| 238 | 
            -
             | 
| 239 | 
            -
             | 
| 240 | 
            -
             | 
| 241 | 
            -
             | 
| 242 | 
            -
             | 
| 243 | 
            -
             | 
| 244 | 
            -
                  def collections_query_kwargs
         | 
| 245 | 
            -
                    query_kwargs.merge(
         | 
| 246 | 
            -
                      fq: [
         | 
| 247 | 
            -
                        %(#{solr_name(work_identifier)}:("#{complete_entry_identifiers.join('" OR "')}")),
         | 
| 248 | 
            -
                        "has_model_ssim:Collection"
         | 
| 249 | 
            -
                      ],
         | 
| 250 | 
            -
                      fl: 'id'
         | 
| 251 | 
            -
                    )
         | 
| 248 | 
            +
                  def works
         | 
| 249 | 
            +
                    @works ||= ParserExportRecordSet.in_batches(complete_entry_identifiers) do |ids|
         | 
| 250 | 
            +
                      ActiveFedora::SolrService.query(
         | 
| 251 | 
            +
                        extra_filters.to_s,
         | 
| 252 | 
            +
                        **query_kwargs.merge(
         | 
| 253 | 
            +
                          fq: [
         | 
| 254 | 
            +
                            %(#{solr_name(work_identifier)}:("#{ids.join('" OR "')}")),
         | 
| 255 | 
            +
                            "has_model_ssim:(#{Bulkrax.curation_concerns.join(' OR ')})"
         | 
| 256 | 
            +
                          ],
         | 
| 257 | 
            +
                          fl: 'id'
         | 
| 258 | 
            +
                        )
         | 
| 259 | 
            +
                      )
         | 
| 260 | 
            +
                    end
         | 
| 252 261 | 
             
                  end
         | 
| 253 262 |  | 
| 254 | 
            -
                  def  | 
| 255 | 
            -
                     | 
| 263 | 
            +
                  def collections
         | 
| 264 | 
            +
                    @collections ||= ParserExportRecordSet.in_batches(complete_entry_identifiers) do |ids|
         | 
| 265 | 
            +
                      ActiveFedora::SolrService.query(
         | 
| 266 | 
            +
                        "has_model_ssim:Collection #{extra_filters}",
         | 
| 267 | 
            +
                        **query_kwargs.merge(
         | 
| 268 | 
            +
                          fq: [
         | 
| 269 | 
            +
                            %(#{solr_name(work_identifier)}:("#{ids.join('" OR "')}")),
         | 
| 270 | 
            +
                            "has_model_ssim:Collection"
         | 
| 271 | 
            +
                          ],
         | 
| 272 | 
            +
                          fl: "id"
         | 
| 273 | 
            +
                        )
         | 
| 274 | 
            +
                      )
         | 
| 275 | 
            +
                    end
         | 
| 256 276 | 
             
                  end
         | 
| 257 277 |  | 
| 258 278 | 
             
                  # This is an exception; we don't know how many candidate file sets there might be.  So we will instead
         | 
| @@ -260,21 +280,18 @@ module Bulkrax | |
| 260 280 | 
             
                  #
         | 
| 261 281 | 
             
                  # @see Bulkrax::ParserExportRecordSet::Base#file_sets
         | 
| 262 282 | 
             
                  def file_sets
         | 
| 263 | 
            -
                    @file_sets ||=  | 
| 264 | 
            -
             | 
| 265 | 
            -
             | 
| 266 | 
            -
             | 
| 267 | 
            -
             | 
| 268 | 
            -
             | 
| 269 | 
            -
             | 
| 270 | 
            -
             | 
| 271 | 
            -
             | 
| 272 | 
            -
             | 
| 273 | 
            -
             | 
| 274 | 
            -
             | 
| 275 | 
            -
             | 
| 276 | 
            -
                  def file_sets_query
         | 
| 277 | 
            -
                    extra_filters
         | 
| 283 | 
            +
                    @file_sets ||= ParserExportRecordSet.in_batches(complete_entry_identifiers) do |ids|
         | 
| 284 | 
            +
                      ActiveFedora::SolrService.query(
         | 
| 285 | 
            +
                        extra_filters,
         | 
| 286 | 
            +
                        query_kwargs.merge(
         | 
| 287 | 
            +
                          fq: [
         | 
| 288 | 
            +
                            %(#{solr_name(work_identifier)}:("#{ids.join('" OR "')}")),
         | 
| 289 | 
            +
                            "has_model_ssim:#{Bulkrax.file_model_class}"
         | 
| 290 | 
            +
                          ],
         | 
| 291 | 
            +
                          fl: 'id'
         | 
| 292 | 
            +
                        )
         | 
| 293 | 
            +
                      )
         | 
| 294 | 
            +
                    end
         | 
| 278 295 | 
             
                  end
         | 
| 279 296 | 
             
                end
         | 
| 280 297 | 
             
              end
         | 
| @@ -10,7 +10,9 @@ | |
| 10 10 | 
             
                    <%= render 'form', importer: @importer, form: form %>
         | 
| 11 11 | 
             
                    <div class="panel-footer">
         | 
| 12 12 | 
             
                      <div class='pull-right'>
         | 
| 13 | 
            -
                         | 
| 13 | 
            +
                        <button type="button" class="btn btn-primary" data-toggle="modal" data-target="#bulkraxModal">
         | 
| 14 | 
            +
                          Update Importer
         | 
| 15 | 
            +
                        </button>
         | 
| 14 16 | 
             
                        <%= render 'edit_form_buttons', form: form %>
         | 
| 15 17 | 
             
                       <% cancel_path = form.object.persisted? ? importer_path(form.object) : importers_path %>
         | 
| 16 18 | 
             
                        | <%= link_to t('.cancel'), cancel_path, class: 'btn btn-default ' %>
         | 
| @@ -19,4 +21,4 @@ | |
| 19 21 | 
             
                  <% end %>
         | 
| 20 22 | 
             
                </div>
         | 
| 21 23 | 
             
              </div>
         | 
| 22 | 
            -
            </div>
         | 
| 24 | 
            +
            </div>
         | 
| @@ -9,8 +9,10 @@ | |
| 9 9 | 
             
                    <%= render 'form', importer: @importer, form: form %>
         | 
| 10 10 | 
             
                    <div class="panel-footer">
         | 
| 11 11 | 
             
                      <div class='pull-right'>
         | 
| 12 | 
            -
                         | 
| 13 | 
            -
             | 
| 12 | 
            +
                        <% if ENV['SHOW_CREATE_AND_VALIDATE'] == 'true' %>
         | 
| 13 | 
            +
                          <%= form.button :submit, value: 'Create and Validate', class: 'btn btn-primary' %>
         | 
| 14 | 
            +
                          |
         | 
| 15 | 
            +
                        <% end  %>
         | 
| 14 16 | 
             
                        <%= form.button :submit, value: 'Create and Import', class: 'btn btn-primary' %>
         | 
| 15 17 | 
             
                        |
         | 
| 16 18 | 
             
                        <%= form.button :submit, value: 'Create', class: 'btn btn-primary' %>
         | 
| @@ -1,10 +1,12 @@ | |
| 1 1 | 
             
            <% if current_ability.can_import_works? %>
         | 
| 2 | 
            -
              <%= menu.nav_link(bulkrax.importers_path | 
| 2 | 
            +
              <%= menu.nav_link(bulkrax.importers_path,
         | 
| 3 | 
            +
                                title: t('bulkrax.admin.sidebar.importers')) do %>
         | 
| 3 4 | 
             
                <span class="fa fa-cloud-upload" aria-hidden="true"></span> <span class="sidebar-action-text"><%= t('bulkrax.admin.sidebar.importers') %></span>
         | 
| 4 5 | 
             
              <% end %>
         | 
| 5 6 | 
             
            <% end %>
         | 
| 6 7 | 
             
            <% if current_ability.can_export_works? %>
         | 
| 7 | 
            -
              <%= menu.nav_link(bulkrax.exporters_path | 
| 8 | 
            +
              <%= menu.nav_link(bulkrax.exporters_path,
         | 
| 9 | 
            +
                                title: t('bulkrax.admin.sidebar.exporters')) do %>
         | 
| 8 10 | 
             
                <span class="fa fa-cloud-download" aria-hidden="true"></span> <span class="sidebar-action-text"><%= t('bulkrax.admin.sidebar.exporters') %></span>
         | 
| 9 11 | 
             
              <% end %>
         | 
| 10 12 | 
             
            <% end %>
         | 
| @@ -3,14 +3,16 @@ | |
| 3 3 | 
             
            <%= menu.nav_link(hyrax.my_collections_path,
         | 
| 4 4 | 
             
                              class: "nav-link",
         | 
| 5 5 | 
             
                              onclick: "dontChangeAccordion(event);",
         | 
| 6 | 
            -
                              also_active_for: hyrax.dashboard_collections_path | 
| 6 | 
            +
                              also_active_for: hyrax.dashboard_collections_path,
         | 
| 7 | 
            +
                              title: t('hyrax.admin.sidebar.collections')) do %>
         | 
| 7 8 | 
             
              <span class="fa fa-folder-open" aria-hidden="true"></span> <span class="sidebar-action-text"><%= t('hyrax.admin.sidebar.collections') %></span>
         | 
| 8 9 | 
             
            <% end %>
         | 
| 9 10 |  | 
| 10 11 | 
             
            <%= menu.nav_link(hyrax.my_works_path,
         | 
| 11 12 | 
             
                              class: "nav-link",
         | 
| 12 13 | 
             
                              onclick: "dontChangeAccordion(event);",
         | 
| 13 | 
            -
                              also_active_for: hyrax.dashboard_works_path | 
| 14 | 
            +
                              also_active_for: hyrax.dashboard_works_path,
         | 
| 15 | 
            +
                              title: t('hyrax.admin.sidebar.works')) do %>
         | 
| 14 16 | 
             
              <span class="fa fa-file" aria-hidden="true"></span> <span class="sidebar-action-text"><%= t('hyrax.admin.sidebar.works') %></span>
         | 
| 15 17 | 
             
            <% end %>
         | 
| 16 18 |  | 
| @@ -0,0 +1,14 @@ | |
| 1 | 
            +
            class AddIndicesToBulkrax < ActiveRecord::Migration[5.1]
         | 
| 2 | 
            +
              def change
         | 
| 3 | 
            +
                add_index :bulkrax_entries, :identifier
         | 
| 4 | 
            +
                add_index :bulkrax_entries, :type
         | 
| 5 | 
            +
                add_index :bulkrax_entries, [:importerexporter_id, :importerexporter_type], name: 'bulkrax_entries_importerexporter_idx'
         | 
| 6 | 
            +
             | 
| 7 | 
            +
                add_index :bulkrax_pending_relationships, :parent_id
         | 
| 8 | 
            +
                add_index :bulkrax_pending_relationships, :child_id
         | 
| 9 | 
            +
             | 
| 10 | 
            +
                add_index :bulkrax_statuses, [:statusable_id, :statusable_type], name: 'bulkrax_statuses_statusable_idx'
         | 
| 11 | 
            +
                add_index :bulkrax_statuses, [:runnable_id, :runnable_type], name: 'bulkrax_statuses_runnable_idx'
         | 
| 12 | 
            +
                add_index :bulkrax_statuses, :error_class
         | 
| 13 | 
            +
              end
         | 
| 14 | 
            +
            end
         | 
    
        data/lib/bulkrax/version.rb
    CHANGED
    
    
    
        data/lib/bulkrax.rb
    CHANGED
    
    | @@ -34,6 +34,15 @@ module Bulkrax | |
| 34 34 | 
             
                              :required_elements,
         | 
| 35 35 | 
             
                              :reserved_properties,
         | 
| 36 36 | 
             
                              :server_name
         | 
| 37 | 
            +
             | 
| 38 | 
            +
                attr_writer :use_locking
         | 
| 39 | 
            +
             | 
| 40 | 
            +
                def use_locking
         | 
| 41 | 
            +
                  return @use_locking if defined?(@use_locking)
         | 
| 42 | 
            +
             | 
| 43 | 
            +
                  ENV.key?("REDIS_HOST")
         | 
| 44 | 
            +
                end
         | 
| 45 | 
            +
                alias use_locking? use_locking
         | 
| 37 46 | 
             
              end
         | 
| 38 47 |  | 
| 39 48 | 
             
              def config
         | 
| @@ -87,7 +96,10 @@ module Bulkrax | |
| 87 96 | 
             
                             :reserved_properties,
         | 
| 88 97 | 
             
                             :reserved_properties=,
         | 
| 89 98 | 
             
                             :server_name,
         | 
| 90 | 
            -
                             :server_name | 
| 99 | 
            +
                             :server_name=,
         | 
| 100 | 
            +
                             :use_locking,
         | 
| 101 | 
            +
                             :use_locking=,
         | 
| 102 | 
            +
                             :use_locking?
         | 
| 91 103 |  | 
| 92 104 | 
             
              config do |conf|
         | 
| 93 105 | 
             
                conf.parsers = [
         | 
| @@ -7,8 +7,8 @@ Bulkrax.setup do |config| | |
| 7 7 | 
             
              # ]
         | 
| 8 8 |  | 
| 9 9 | 
             
              # WorkType to use as the default if none is specified in the import
         | 
| 10 | 
            -
              # Default is the first returned by Hyrax.config.curation_concerns
         | 
| 11 | 
            -
              # config.default_work_type = MyWork
         | 
| 10 | 
            +
              # Default is the first returned by Hyrax.config.curation_concerns, stringified
         | 
| 11 | 
            +
              # config.default_work_type = "MyWork"
         | 
| 12 12 |  | 
| 13 13 | 
             
              # Factory Class to use when generating and saving objects
         | 
| 14 14 | 
             
              config.object_factory = Bulkrax::ObjectFactory
         | 
    
        metadata
    CHANGED
    
    | @@ -1,14 +1,14 @@ | |
| 1 1 | 
             
            --- !ruby/object:Gem::Specification
         | 
| 2 2 | 
             
            name: bulkrax
         | 
| 3 3 | 
             
            version: !ruby/object:Gem::Version
         | 
| 4 | 
            -
              version: 5. | 
| 4 | 
            +
              version: 5.3.0
         | 
| 5 5 | 
             
            platform: ruby
         | 
| 6 6 | 
             
            authors:
         | 
| 7 7 | 
             
            - Rob Kaufman
         | 
| 8 8 | 
             
            autorequire:
         | 
| 9 9 | 
             
            bindir: bin
         | 
| 10 10 | 
             
            cert_chain: []
         | 
| 11 | 
            -
            date: 2023- | 
| 11 | 
            +
            date: 2023-08-02 00:00:00.000000000 Z
         | 
| 12 12 | 
             
            dependencies:
         | 
| 13 13 | 
             
            - !ruby/object:Gem::Dependency
         | 
| 14 14 | 
             
              name: rails
         | 
| @@ -52,6 +52,20 @@ dependencies: | |
| 52 52 | 
             
                - - ">="
         | 
| 53 53 | 
             
                  - !ruby/object:Gem::Version
         | 
| 54 54 | 
             
                    version: '0'
         | 
| 55 | 
            +
            - !ruby/object:Gem::Dependency
         | 
| 56 | 
            +
              name: dry-monads
         | 
| 57 | 
            +
              requirement: !ruby/object:Gem::Requirement
         | 
| 58 | 
            +
                requirements:
         | 
| 59 | 
            +
                - - "~>"
         | 
| 60 | 
            +
                  - !ruby/object:Gem::Version
         | 
| 61 | 
            +
                    version: 1.4.0
         | 
| 62 | 
            +
              type: :runtime
         | 
| 63 | 
            +
              prerelease: false
         | 
| 64 | 
            +
              version_requirements: !ruby/object:Gem::Requirement
         | 
| 65 | 
            +
                requirements:
         | 
| 66 | 
            +
                - - "~>"
         | 
| 67 | 
            +
                  - !ruby/object:Gem::Version
         | 
| 68 | 
            +
                    version: 1.4.0
         | 
| 55 69 | 
             
            - !ruby/object:Gem::Dependency
         | 
| 56 70 | 
             
              name: iso8601
         | 
| 57 71 | 
             
              requirement: !ruby/object:Gem::Requirement
         | 
| @@ -238,6 +252,34 @@ dependencies: | |
| 238 252 | 
             
                - - ">="
         | 
| 239 253 | 
             
                  - !ruby/object:Gem::Version
         | 
| 240 254 | 
             
                    version: '0'
         | 
| 255 | 
            +
            - !ruby/object:Gem::Dependency
         | 
| 256 | 
            +
              name: redis
         | 
| 257 | 
            +
              requirement: !ruby/object:Gem::Requirement
         | 
| 258 | 
            +
                requirements:
         | 
| 259 | 
            +
                - - "~>"
         | 
| 260 | 
            +
                  - !ruby/object:Gem::Version
         | 
| 261 | 
            +
                    version: '4.2'
         | 
| 262 | 
            +
              type: :development
         | 
| 263 | 
            +
              prerelease: false
         | 
| 264 | 
            +
              version_requirements: !ruby/object:Gem::Requirement
         | 
| 265 | 
            +
                requirements:
         | 
| 266 | 
            +
                - - "~>"
         | 
| 267 | 
            +
                  - !ruby/object:Gem::Version
         | 
| 268 | 
            +
                    version: '4.2'
         | 
| 269 | 
            +
            - !ruby/object:Gem::Dependency
         | 
| 270 | 
            +
              name: psych
         | 
| 271 | 
            +
              requirement: !ruby/object:Gem::Requirement
         | 
| 272 | 
            +
                requirements:
         | 
| 273 | 
            +
                - - "~>"
         | 
| 274 | 
            +
                  - !ruby/object:Gem::Version
         | 
| 275 | 
            +
                    version: '3.3'
         | 
| 276 | 
            +
              type: :development
         | 
| 277 | 
            +
              prerelease: false
         | 
| 278 | 
            +
              version_requirements: !ruby/object:Gem::Requirement
         | 
| 279 | 
            +
                requirements:
         | 
| 280 | 
            +
                - - "~>"
         | 
| 281 | 
            +
                  - !ruby/object:Gem::Version
         | 
| 282 | 
            +
                    version: '3.3'
         | 
| 241 283 | 
             
            description: Bulkrax is a batteries included importer for Samvera applications. It
         | 
| 242 284 | 
             
              currently includes support for OAI-PMH (DC and Qualified DC) and CSV out of the
         | 
| 243 285 | 
             
              box. It is also designed to be extensible, allowing you to easily add new importers
         | 
| @@ -258,6 +300,7 @@ files: | |
| 258 300 | 
             
            - app/assets/javascripts/bulkrax/entries.js
         | 
| 259 301 | 
             
            - app/assets/javascripts/bulkrax/exporters.js
         | 
| 260 302 | 
             
            - app/assets/javascripts/bulkrax/importers.js.erb
         | 
| 303 | 
            +
            - app/assets/javascripts/bulkrax/navtabs.js.erb
         | 
| 261 304 | 
             
            - app/assets/stylesheets/bulkrax/accordion.scss
         | 
| 262 305 | 
             
            - app/assets/stylesheets/bulkrax/application.css
         | 
| 263 306 | 
             
            - app/assets/stylesheets/bulkrax/coderay.scss
         | 
| @@ -395,6 +438,7 @@ files: | |
| 395 438 | 
             
            - db/migrate/20220412233954_add_include_thumbnails_to_bulkrax_exporters.rb
         | 
| 396 439 | 
             
            - db/migrate/20220413180915_add_generated_metadata_to_bulkrax_exporters.rb
         | 
| 397 440 | 
             
            - db/migrate/20220609001128_rename_bulkrax_importer_run_to_importer_run.rb
         | 
| 441 | 
            +
            - db/migrate/20230608153601_add_indices_to_bulkrax.rb
         | 
| 398 442 | 
             
            - lib/bulkrax.rb
         | 
| 399 443 | 
             
            - lib/bulkrax/engine.rb
         | 
| 400 444 | 
             
            - lib/bulkrax/entry_spec_helper.rb
         | 
| @@ -427,7 +471,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement | |
| 427 471 | 
             
                - !ruby/object:Gem::Version
         | 
| 428 472 | 
             
                  version: '0'
         | 
| 429 473 | 
             
            requirements: []
         | 
| 430 | 
            -
            rubygems_version: 3. | 
| 474 | 
            +
            rubygems_version: 3.1.6
         | 
| 431 475 | 
             
            signing_key:
         | 
| 432 476 | 
             
            specification_version: 4
         | 
| 433 477 | 
             
            summary: Import and export tool for Hyrax and Hyku
         |