iiif_print 1.0.0 → 1.1.0

data/README.md

@@ -35,9 +35,9 @@ IiifPrint supports:
 * OCR keyword match highlighting
 * viewer with page navigation and deep zooming
 * splitting of PDFs to LZW compressed TIFFs for viewing
-* configuring how the manifest canvases are sorted in the viewer
 * adding metadata fields to the manifest with faceted search links and external links
 * excluding specified work types to be found in the catalog search
+* external IIIF image urls that work with services such as serverless-iiif or cantaloup
 
 A complete list of features can be found [here](https://github.com/scientist-softserv/iiif_print/wiki/Features-List).
 
@@ -86,15 +86,39 @@ IiifPrint easily integrates with your Hyrax 2.x applications.
 * In `config/routes.rb`, it adds `concerns :iiif_search` in the `resources :solr_documents` block
 * Adds `config/initializers/iiif_print.rb`
 * Adds three migrations, `CreateIiifPrintDerivativeAttachments`, `CreateIiifPrintIngestFileRelations`, and `CreateIiifPrintPendingRelationships`
-* In `solr/conf/schema.xml`, it adds Blacklight IIIF Search autocomplete config
-* In `solr/conf/solrconfig.xml`, it adds Blacklight IIIF Search autocomplete config
-* Adds `solr/lib/solr-tokenizing_suggester-7.x.jar`
 
 (It may be helpful to run `git diff` after installation to see all the changes made by the installer.)
 
+## Catalog to Universal Viewer search:
+To enable a feature where the UV automatically picks up the search from the catalog, do the following:
+* Add `highlight: urlDataProvider.get('q'),` into your uv.html in the `<script>` section.
+```js
+uv = createUV('#uv', {
+    root: '.',
+    iiifResourceUri: urlDataProvider.get('manifest'),
+    configUri: 'uv-config.json',
+    collectionIndex: Number(urlDataProvider.get('c', 0)),
+    manifestIndex: Number(urlDataProvider.get('m', 0)),
+    sequenceIndex: Number(urlDataProvider.get('s', 0)),
+    canvasIndex: Number(urlDataProvider.get('cv', 0)),
+    rangeId: urlDataProvider.get('rid', 0),
+    rotation: Number(urlDataProvider.get('r', 0)),
+    xywh: urlDataProvider.get('xywh', ''),
+    embedded: true,
+    highlight: urlDataProvider.get('q'), // <-- here's a good spot
+    locales: formattedLocales
+}, urlDataProvider);
+```
+
+* Make sure to remove your application's `app/helpers/hyrax/iiif_helper.rb` and `app/views/hyrax/base/iiif_viewers/_universal_viewer.html.erb` (if exists)
+
 ## Configuration to enable IiifPrint features
 **NOTE: WorkTypes and models are used synonymously here.**
 
+### IIIF URL configuration
+
+If you set EXTERNAL_IIIF_URL in your environment, then IiifPrint will use that URL as the root for your IIIF URLs. It will also switch from using the file set ID to using the SHA1 of the file as the identifier. This enables using serverless_iiif or Cantaloupe (refered to as the service) by pointing the service to the same S3 bucket that FCREPO writes the uploaded files to. By setting it up that way you do not need the service to connect to FCREPO or Hyrax at all, both natively support connecting to an S3 bucket to get their data.
+
 ### Model level configurations
 
 In `app/models/{work_type}.rb` add `include IiifPrint.model_configuration` to any work types which require IiifPrint processing features (such as PDF splitting or OCR derivatives). See [lib/iiif_print.rb](./lib/iiif_print.rb) for details on configuration options.
@@ -126,10 +150,6 @@ IiifPrint.config do |config|
   # Add configurable solr field key for searching, default key is: 'human_readable_type_sim' if
   # another key is used, make sure to adjust the config.excluded_model_name_solr_field_values to match
   config.excluded_model_name_solr_field_key = 'some_solr_field_key'
-
-  # Configure how the manifest sorts the canvases, by default it sorts by `:title`, but a different
-  # model property may be desired such as :date_published
-  config.sort_iiif_manifest_canvases_by = :date_published
 end
 ```
 
@@ -146,7 +166,7 @@ TO ENABLE OCR Search (from the UV and catalog search)
       }
     end
 ```
-* Set `config.search_builder_class = IiifPrint::CatalogSearchBuilder` to remove works from the catalog search results if `is_child_bsi: true` 
+* Set `config.search_builder_class = IiifPrint::CatalogSearchBuilder` to remove works from the catalog search results if `is_child_bsi: true`
 * Ensure that all text search is configured in default_solr_params config block:
 ```rb
     config.default_solr_params = {
@@ -156,6 +176,75 @@ TO ENABLE OCR Search (from the UV and catalog search)
     }
 ```
 
+To remove child works from recent works on homepage
+### homepage_controller.rb
+* In the HomepageController, change the search_builder_class to remove works from recent_documents if `is_child_bsi: true`
+```rb
+    require "iiif_print/homepage_search_builder"
+
+    def search_builder_class
+      IiifPrint::HomepageSearchBuilder
+    end
+```
+
+### Skipping Certain File Suffixes for PDF Splitting
+
+By default when a work is configured for splitting PDFs, we will split all PDFs.  However, in some cases you don't want to split based on the file name's suffix.  In that case, configure code as follows:
+
+```ruby
+IiifPrint.config do |config|
+  config.skip_splitting_pdf_files_that_end_with_these_texts = ['.reader.pdf']
+end
+```
+
+### Derivative Rodeo Configuration
+
+The Derivative Rodeo is used in two ways:
+
+- Configuring the `Hyrax::DerivativeService` by adding `IiifPrint::DerivativeRodeoService`
+- Enable Derivative Rodeo PDF Splitting service by `IiifPrint.model_configuration`
+
+#### Configuring Hyrax::Derivative
+
+In the application initializer:
+
+```ruby
+      Hyrax::DerivativeService.services = [
+        IiifPrint::DerivativeRodeoService,
+        Hyrax::FileSetDerivativesService]
+```
+
+#### Enabling Derivative Rodeo PDF Splitting
+
+The [IiifPrint.model\_configuration  method](./lib/iiif_print.rb) allows for specifying the `pdf\_splitter\_service` as below:
+
+```ruby
+class Book < ActiveFedora::Base
+  include IiifPrint.model_configuration(
+            pdf_splitter_service: IiifPrint::SplitPdfs::DerivativeRodeoSplitter
+          )
+end
+```
+
+#### Pre-Process Location
+
+The [DerivativeRodeo](https://github.com/scientist-softserv/derivative_rodeo) allows for specifying a location where you've done pre-processing (e.g. you ran splitting and derivative generation in AWS's Lambda).
+
+By default the preprocess location is S3, as that is where SoftServ has been running pre-processing.  However that default may not be adequate for local development.
+
+#### Conditional Derivative Generation
+
+The [IiifPrint::DerivativeRodeoService][./app/services/iiif_print/derivative_rodeo_service.rb] provides a means of specifying the derivatives to generate via two configuration points:
+
+- `IiifPrint::DerivativeRodeoService.named_derivatives_and_generators_by_type`
+- `IiifPrint::DerivativeRodeoService.named_derivatives_and_generators_filter`
+
+In the case of `named_derivatives_and_generators_by_type`, we're saying all mime categories will generate these derivatives.
+
+In the case of `named_derivatives_and_generators_filter`, we're providing a point where we can specify for each file_set and filename the specific derivatives to accept/reject/append to the named derivative generation.
+
+See their examples for further configuration guidance.
+
 # Ingesting Content
 
 IiifPrint supports a range of different ingest workflows:

data/Rakefile

@@ -35,4 +35,10 @@ end
 Dir.glob('tasks/*.rake').each { |r| import r }
 Dir.glob('lib/tasks/*.rake').each { |r| import r }
 
+# Adding the copy_authorities here so it runs the same in CI
+desc "Generate the engine_cart, copy authorities, and run tests"
+task prepare_and_run_tests: ['engine_cart:generate', 'engine_cart:copy_authorities'] do
+  puts "Running CI tests"
+end
+
 task default: :ci

data/app/actors/iiif_print/actors/cleanup_file_sets_actor_decorator.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+# override Hyrax to remove splitting upon work delete
+module IiifPrint
+  module Actors
+    # Responsible for removing FileSets related to the given curation concern.
+    module CleanupFileSetsActorDecorator
+      # @param [Hyrax::Actors::Environment] env
+      # @return [Boolean] true if destroy was successful
+      def destroy(env)
+        file_sets = env.curation_concern.file_sets
+        file_sets.each do |file_set|
+          # we destroy the children before the file_set, because we need the parent relationship
+          IiifPrint::SplitPdfs::DestroyPdfChildWorksService.conditionally_destroy_spawned_children_of(
+            file_set: file_set,
+            work: env.curation_concern
+          )
+        end
+        # and now back to your regularly scheduled programming
+        super
+      end
+    end
+  end
+end

data/app/actors/iiif_print/actors/file_set_actor_decorator.rb

@@ -1,6 +1,9 @@
 # frozen_string_literal: true
 
-# override to add PDF splitting for file sets
+# override to add PDF splitting for file sets and remove splitting upon fileset delete
+
+# Depending on whether we have an uploaded file or a remote url, the sequence of calling
+# attach_to_work and create_content will switch.
 module IiifPrint
   module Actors
     module FileSetActorDecorator
@@ -9,48 +12,47 @@ module IiifPrint
         super
 
         if from_url
-          # we have everything we need... queue the job
-          parent = parent_for(file_set: @file_set)
-
-          if service.iiif_print_split?(work: parent) && service.pdfs?(paths: [file_set.import_url])
-            service.queue_job(
-              work: parent,
-              file_locations: [file.path],
-              user: @user,
-              admin_set_id: parent.admin_set_id
-            )
-          end
+          # in this case, the file that came in is a temp file, and we need to use the actual file.
+          # the file was attached to the file_set in Hyrax::ImportUrlJob so we can just access it.
+          args = { file_set: file_set, file: file_set.files.first, import_url: file_set.import_url, user: @user }
+          returned_value = service.conditionally_enqueue(**args)
+          Rails.logger.info("Result of #{returned_value} for conditional enqueueing of #{args.inspect}")
+          true
         else
           # we don't have the parent yet... save the paths for later use
-          @pdf_paths = service.pdf_paths(files: [file.id.to_s])
+          @file = file
         end
       end
 
-      # Prior to Hyrax v3.1.0, this method did not exist
-      # @param file_set [FileSet]
-      # @return [ActiveFedora::Base]
-      def parent_for(file_set:)
-        file_set.parent
-      end
-
       # Override to add PDF splitting
       def attach_to_work(work, file_set_params = {})
         # Locks to ensure that only one process is operating on the list at a time.
         super
 
-        return if @pdf_paths.blank?
-        return unless service.iiif_print_split?(work: work)
-        service.queue_job(
-          work: work,
-          file_locations: @pdf_paths,
-          user: @user,
-          admin_set_id: work.admin_set_id
-        )
+        # when we are importing a remote_url, this method is called before the file is attached.
+        # We want to short-circuit the process and prevent unnecessarily confusing logging.
+        return unless @file
+
+        args = { file_set: file_set, work: work, file: @file, user: @user }
+        returned_value = service.conditionally_enqueue(**args)
+        Rails.logger.info("Result of #{returned_value} for conditional enqueueing of #{args.inspect}")
+        true
       end
 
       def service
         IiifPrint::SplitPdfs::ChildWorkCreationFromPdfService
       end
+
+      # Clean up children when removing the fileset
+      def destroy
+        # we destroy the children before the file_set, because we need the parent relationship
+        IiifPrint::SplitPdfs::DestroyPdfChildWorksService.conditionally_destroy_spawned_children_of(
+          file_set: file_set,
+          work: file_set.parent
+        )
+        # and now back to your regularly scheduled programming
+        super
+      end
     end
   end
 end

data/app/controllers/iiif_print/split_pdfs_controller.rb

@@ -0,0 +1,38 @@
+module IiifPrint
+  # Responsible for coordinating the request to resplit a PDF.
+  class SplitPdfsController < ApplicationController
+    before_action :authenticate_user!
+
+    def create
+      @file_set = FileSet.where(id: params[:file_set_id]).first
+      authorize_create_split_request!(@file_set)
+      IiifPrint::Jobs::RequestSplitPdfJob.perform_later(file_set: @file_set, user: current_user)
+      respond_to do |wants|
+        wants.html { redirect_to polymorphic_path([main_app, @file_set]), notice: t("iiif_print.file_set.split_submitted", id: @file_set.id) }
+        wants.json { render json: { id: @file_set.id, to_param: @file_set.to_param }, status: :ok }
+      end
+    end
+
+    private
+
+    ##
+    # @param file_set [FileSet]
+    def authorize_create_split_request!(file_set)
+      # NOTE: Duplicates logic of Hyrax: https://github.com/samvera/hyrax/blob/b334e186e77691d7da8ed59ff27f091be1c2a700/app/controllers/hyrax/file_sets_controller.rb#L234-L241
+      #
+      # Namely if we don't have a file_set we need not proceed.
+      raise CanCan::AccessDenied unless file_set
+
+      ##
+      # Rely on CanCan's authorize! method.  We could add the :split_pdf action to the ability
+      # class.  But we're pigging backing on the idea that you can do this if you can edit the work.
+      authorize!(:edit, file_set)
+      raise "Expected #{file_set.class} ID=#{file_set.id} #to_param=#{file_set.to_param} to be a PDF.  Instead found mime_type of #{file_set.mime_type}." unless file_set.pdf?
+
+      work = IiifPrint.parent_for(file_set)
+      raise WorkNotConfiguredToSplitFileSetError.new(file_set: file_set, work: work) unless work&.iiif_print_config&.pdf_splitter_job&.presence
+
+      true
+    end
+  end
+end

data/app/helpers/iiif_print/iiif_helper_decorator.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+# OVERRIDE Hyrax v2.9.6 add #uv_search_param
+
+module IiifPrint
+  module IiifHelperDecorator
+    def iiif_viewer_display(work_presenter, locals = {})
+      render iiif_viewer_display_partial(work_presenter),
+             locals.merge(presenter: work_presenter)
+    end
+
+    def iiif_viewer_display_partial(work_presenter)
+      'hyrax/base/iiif_viewers/' + work_presenter.iiif_viewer.to_s
+    end
+
+    def universal_viewer_base_url
+      "#{request&.base_url}#{IiifPrint.config.uv_base_path}"
+    end
+
+    def universal_viewer_config_url
+      "#{request&.base_url}#{IiifPrint.config.uv_config_path}"
+    end
+
+    # Extract query param from search
+    def uv_search_param
+      search_params = current_search_session.try(:query_params) || {}
+      q = search_params['q'].presence || ''
+
+      "&q=#{url_encode(q)}" if q.present?
+    end
+  end
+end

data/app/helpers/iiif_print/iiif_print_helper_behavior.rb

@@ -0,0 +1,23 @@
+module IiifPrint::IiifPrintHelperBehavior
+  ##
+  # print the ocr snippets. if more than one, separate with <br/>
+  #
+  # @param options [Hash] options hash provided by Blacklight
+  # @return [String] snippets HTML to be rendered
+  # rubocop:disable Rails/OutputSafety
+  def render_ocr_snippets(options = {})
+    snippets = options[:value]
+    return if snippets.blank?
+
+    snippets_content = [content_tag('div',
+                                    "... #{snippets.first} ...".html_safe,
+                                    class: 'ocr_snippet first_snippet')]
+    if snippets.length > 1
+      snippets_content << render(partial: 'catalog/snippets_more',
+                                 locals: { snippets: snippets.drop(1),
+                                           options: options })
+    end
+    snippets_content.join("\n").html_safe
+  end
+  # rubocop:enable Rails/OutputSafety
+end

data/app/helpers/iiif_print_helper.rb

@@ -41,24 +41,4 @@ module IiifPrintHelper
     end
     hl_matches.uniq.sort.join(' ')
   end
-
-  ##
-  # print the ocr snippets. if more than one, separate with <br/>
-  #
-  # @param options [Hash] options hash provided by Blacklight
-  # @return [String] snippets HTML to be rendered
-  # rubocop:disable Rails/OutputSafety
-  def render_ocr_snippets(options = {})
-    snippets = options[:value]
-    snippets_content = [content_tag('div',
-                                    "... #{snippets.first} ...".html_safe,
-                                    class: 'ocr_snippet first_snippet')]
-    if snippets.length > 1
-      snippets_content << render(partial: 'catalog/snippets_more',
-                                 locals: { snippets: snippets.drop(1),
-                                           options: options })
-    end
-    snippets_content.join("\n").html_safe
-  end
-  # rubocop:enable Rails/OutputSafety
 end

data/app/indexers/concerns/iiif_print/child_indexer.rb

@@ -19,15 +19,21 @@ module IiifPrint
           indexer.prepend(self)
           indexer.class_attribute(:iiif_print_lineage_service, default: IiifPrint::LineageService)
         end
-        work_type::GeneratedResourceSchema.send(:include, IiifPrint::SetChildFlag)
+        work_type::GeneratedResourceSchema.send(:include, IiifPrint::SetChildFlag) if work_type.const_defined?(:GeneratedResourceSchema)
       end
     end
 
     def generate_solr_document
       super.tap do |solr_doc|
-        solr_doc['is_child_bsi'] = object.is_child
+        solr_doc['is_child_bsi'] ||= object.is_child
+        solr_doc['split_from_pdf_id_ssi'] ||= object.split_from_pdf_id
         solr_doc['is_page_of_ssim'] = iiif_print_lineage_service.ancestor_ids_for(object)
-        solr_doc['file_set_ids_ssim'] = iiif_print_lineage_service.descendent_file_set_ids_for(object)
+
+        # Due to a long-standing hack in Hyrax, the file_set_ids_ssim contains both file_set_ids and
+        # child work ids.
+        #
+        # See https://github.com/samvera/hyrax/blob/2b807fe101176d594129ef8a8fe466d3d03a372b/app/indexers/hyrax/work_indexer.rb#L15-L18
+        solr_doc['file_set_ids_ssim'] = iiif_print_lineage_service.descendent_member_ids_for(object)
       end
     end
   end

data/app/indexers/concerns/iiif_print/file_set_indexer.rb

@@ -19,11 +19,24 @@ module IiifPrint
         # only UV viewable images should have is_page_of, it is only used for iiif search
         solr_doc['is_page_of_ssim'] = iiif_print_lineage_service.ancestor_ids_for(object) if object.mime_type&.match(/image/)
         # index for full text search
-        text = IiifPrint::Data::WorkDerivatives.data(from: object, of_type: 'txt')
-        text = text.tr("\n", ' ').squeeze(' ')
-        solr_doc['all_text_timv'] = text
-        solr_doc['all_text_tsimv'] = text
+        solr_doc['all_text_timv'] = all_text
+        solr_doc['all_text_tsimv'] = all_text
+        solr_doc['digest_ssim'] = digest_from_content
       end
     end
+
+    private
+
+    def digest_from_content
+      return unless object.original_file
+      object.original_file.digest.first.to_s
+    end
+
+    def all_text
+      text = IiifPrint.config.all_text_generator_function.call(object: object) || ''
+      return text if text.empty?
+
+      text.tr("\n", ' ').squeeze(' ')
+    end
   end
 end

data/app/models/concerns/iiif_print/set_child_flag.rb

@@ -4,6 +4,10 @@ module RDF
   class CustomIsChildTerm < Vocabulary('http://id.loc.gov/vocabulary/identifiers/')
     property 'is_child'
   end
+
+  class FromPdfIdTerm < Vocabulary('http://id.loc.gov/vocabulary/identifiers/')
+    property 'split_from_pdf_id'
+  end
 end
 
 module IiifPrint
@@ -18,6 +22,11 @@ module IiifPrint
               multiple: false do |index|
                 index.as :stored_searchable
               end
+      property :split_from_pdf_id,
+              predicate: ::RDF::FromPdfIdTerm.split_from_pdf_id,
+              multiple: false do |index|
+                index.as :stored_searchable
+              end
     end
 
     def set_children

data/app/models/concerns/iiif_print/solr/document.rb

@@ -19,6 +19,8 @@ module IiifPrint::Solr::Document
   def self.decorate(base)
     base.prepend(self)
     base.send(:attribute, :is_child, Hyrax::SolrDocument::Metadata::Solr::String, 'is_child_bsi')
+    base.send(:attribute, :split_from_pdf_id, Hyrax::SolrDocument::Metadata::Solr::String, 'split_from_pdf_id_ssi')
+    base.send(:attribute, :digest, Hyrax::SolrDocument::Metadata::Solr::String, 'digest_ssim')
 
     # @note These properties came from the newspaper_works gem.  They are configurable.
     base.class_attribute :iiif_print_solr_field_names, default: %w[alternative_title genre
@@ -31,6 +33,10 @@ module IiifPrint::Solr::Document
     base
   end
 
+  def digest_sha1
+    digest[/urn:sha1:([\w]+)/, 1]
+  end
+
   def method_missing(method_name, *args, &block)
     super unless iiif_print_solr_field_names.include? method_name.to_s
     self[::ActiveFedora.index_field_mapper.solr_name(method_name.to_s)]
@@ -44,4 +50,12 @@ module IiifPrint::Solr::Document
   def file_set_ids
     self['file_set_ids_ssim']
   end
+
+  def any_highlighting?
+    response&.[]('highlighting')&.[](id)&.present?
+  end
+
+  def solr_document
+    self
+  end
 end

data/app/models/iiif_print/iiif_search_decorator.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+# OVERRIDE: Blacklight IIIF Search v1.0.0
+# IiifSearchDecorator module extends the functionality of the BlacklightIiifSearch::IiifSearch class
+# by overriding the solr_params method to modify the search query to include the parent's metadata.
+module IiifPrint
+  module IiifSearchDecorator
+    ##
+    # Overrides the solr_params method from BlacklightIiifSearch::IiifSearch to modify the search query.
+    # The method adds an additional filter to the query to include either the object_relation_field OR the
+    # parent document's id and removes the :f parameter from the query.
+    # :object_relation_field refers to the CatalogController's configuration which is typically set to
+    # 'is_page_of_ssim' in the host application which only searches child works by default.
+    #
+    #   config.iiif_search = {
+    #     full_text_field: 'all_text_tsimv',
+    #     object_relation_field: 'is_page_of_ssim',
+    #     supported_params: %w[q page],
+    #     autocomplete_handler: 'iiif_suggest',
+    #     suggester_name: 'iiifSuggester'
+    #   }
+    #
+    # @return [Hash] A hash containing the modified Solr search parameters
+    #
+    def solr_params
+      return { q: 'nil:nil' } unless q
+
+      {
+        q: "#{q} AND (#{iiif_config[:object_relation_field]}:\"#{parent_document.id}\" OR id:\"#{parent_document.id}\")",
+        rows: rows,
+        page: page
+      }
+    end
+  end
+end

data/app/models/iiif_print/iiif_search_response_decorator.rb

@@ -4,14 +4,37 @@ module IiifPrint
     # @see https://github.com/scientist-softserv/louisville-hyku/commit/67467e5cf9fdb755f54419f17d3c24c87032d0af
     def annotation_list
       json_results = super
-      json_results&.[]('resources')&.each do |result_hit|
+
+      # Check and process invalid hit
+      if json_results&.[]('resources')
+        remove_invalid_hit(json_results)
+        add_metadata_match(json_results)
+      end
+
+      json_results
+    end
+
+    def remove_invalid_hit(json_results)
+      resources = json_results['resources']
+      invalid_hit = resources.detect { |resource| resource["on"].include?(IiifPrint::BlacklightIiifSearch::AnnotationDecorator::INVALID_MATCH_TEXT) }
+      return unless invalid_hit
+
+      # Delete invalid hit from resources, remove first hit (which is from the invalid hit), decrement total within
+      resources.delete(invalid_hit)
+      json_results['hits'].shift
+      json_results['within']['total'] -= 1
+    end
+
+    def add_metadata_match(json_results)
+      json_results['resources'].each do |result_hit|
         next if result_hit['resource'].present?
+
+        # Add resource details if not present
         result_hit['resource'] = {
           "@type": "cnt:ContentAsText",
           "chars": "Metadata match, see sidebar for details"
         }
       end
-      json_results
     end
   end
 end

data/app/models/iiif_print/pending_relationship.rb

@@ -3,5 +3,8 @@ module IiifPrint
     validates :parent_id, presence: true
     validates :child_title, presence: true
     validates :child_order, presence: true
+    validates :parent_model, presence: true
+    validates :child_model, presence: true
+    validates :file_id, presence: true
   end
 end