zizia 4.0.3.alpha.01 → 4.0.4.alpha.01

data/db/migrate/20190921153851_add_collection_id_to_csv_import_detail.rb

@@ -0,0 +1,5 @@
+class AddCollectionIdToCsvImportDetail < ActiveRecord::Migration[5.1]
+  def change
+    add_column :zizia_csv_import_details, :collection_id, :string
+  end
+end

data/db/migrate/20190921153933_add_batch_id_to_csv_import_detail.rb

@@ -0,0 +1,5 @@
+class AddBatchIdToCsvImportDetail < ActiveRecord::Migration[5.1]
+  def change
+    add_column :zizia_csv_import_details, :batch_id, :string
+  end
+end

data/db/migrate/20190921154009_add_success_count_to_csv_import_detail.rb

@@ -0,0 +1,5 @@
+class AddSuccessCountToCsvImportDetail < ActiveRecord::Migration[5.1]
+  def change
+    add_column :zizia_csv_import_details, :success_count, :integer
+  end
+end

data/db/migrate/20190921154038_add_failure_count_to_csv_import_detail.rb

@@ -0,0 +1,5 @@
+class AddFailureCountToCsvImportDetail < ActiveRecord::Migration[5.1]
+  def change
+    add_column :zizia_csv_import_details, :failure_count, :integer
+  end
+end

data/db/migrate/20190921154221_add_deduplication_field_to_csv_import_detail.rb

@@ -0,0 +1,5 @@
+class AddDeduplicationFieldToCsvImportDetail < ActiveRecord::Migration[5.1]
+  def change
+    add_column :zizia_csv_import_details, :deduplication_field, :string
+  end
+end

data/db/migrate/20190923182946_add_update_actor_stack_to_csv_import_detail.rb

@@ -0,0 +1,5 @@
+class AddUpdateActorStackToCsvImportDetail < ActiveRecord::Migration[5.1]
+  def change
+    add_column :zizia_csv_import_details, :update_actor_stack, :string
+  end
+end

data/docs/_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-minimal

data/docs/customizing_metadata.md

@@ -0,0 +1,41 @@
+# Customizing metadata import
+
+Often the data you will want to import will not conform exactly to the fields you get from `Hyrax::BasicMetadata`, in which case you will need to make a custom mapper.
+
+Again, let's start with making a test. Make a file called `spec/importers/custom_mapper.rb`. Let's start by describing the CSV we expect to import:
+
+```ruby
+# frozen_string_literal: true
+
+require 'rails_helper'
+
+RSpec.describe CustomMapper do
+  subject(:mapper) { described_class.new }
+
+  let(:metadata) do
+    { "Name.architect" => "Imhotep", # architect
+      "Type.genre" => "Journalism" # genre
+    }
+
+    describe '#fields' do
+    it 'has expected fields' do
+      expect(mapper.fields).to include(
+        :architect,
+        :genre
+      )
+    end
+  end
+```
+From this rspec test, I can conclude that I have two customized metadata fields, `architect` and `genre`. Actually adding custom metadata fields is covered elsewhere (See [this guide](https://samvera.github.io/customize-metadata-generate-work-type.html), for example), so here we are just going to assume that these fields exist already, and are defined as multi-valued strings.
+
+
+* Put a file in `config/initializers/zizia.rb`
+
+It should look like this:
+
+```ruby
+Zizia.config do |config|
+  config.metadata_mapper_class = CustomMapper
+end
+```
+This tells zizia what class to use for metadata mappings. Output is logged to `Rails.logger` and all output is prefixed with `[zizia]` for easy log parsing.

data/docs/index.md

@@ -0,0 +1,113 @@
+# Importing to Hyrax with Zizia
+
+## Getting Started
+The simplest use case is importing content that matches [Hyrax's core and basic metadata fields](http://samvera.github.io/metadata_application_profile.html), plus a few extra fields that let Hyrax know how to display the content properly. At a very high level we're going to:
+1. Write a simple import test first
+1. Get the test passing
+1. Write a rake task to run your import with real data
+
+**Note:** This guide assumes that you already have a working Hyrax instance with at least one work type. If you need help doing that, see [the Hyrax documentation](https://github.com/samvera/hyrax#creating-a-hyrax-based-app). This guide is going to assume we're using an `Image` work type and attaching image files, but should be applicable to any work type or kind of attachment.
+
+### 1. Write a simple import test
+1. Make a directory for your importer: `mkdir app/importers`
+1. Make a directory for your importer tests: `mkdir spec/importers`
+1. Make a directory for your fixture files: `mkdir spec/fixtures/images`
+1. Put three small images in `spec/fixtures/images` In this guide, we're using copyright free images from https://www.pexels.com/, `birds.jpg`, `cat.jpg`, and `dog.jpg`.
+1. Make a directory for your CSV fixture files: `mkdir spec/fixtures/csv_import`
+1. Put a file like this in `spec/fixtures/csv_import/zizia_basic.csv`:
+  ```
+  title,source,visibility
+  "A Cute Dog",https://www.pexels.com/photo/animal-blur-canine-close-up-551628/,open
+  "An Interesting Cat",https://www.pexels.com/photo/person-holding-white-cat-1383397/,open
+  "A Flock of Birds",https://www.pexels.com/photo/animal-avian-beak-birds-203088/,open
+  ```
+1. Make a file called `spec/importers/modular_importer_spec.rb` that contains this:
+  ```ruby
+    # frozen_string_literal: true
+
+    require 'rails_helper'
+
+    RSpec.describe ModularImporter do
+      let(:modular_csv)     { 'spec/fixtures/csv_import/zizia_basic.csv' }
+      let(:user) { ::User.batch_user }
+
+      it "imports a csv" do
+        expect { ModularImporter.new(modular_csv).import }.to change { Work.count }.by 3
+      end
+    end
+  ```
+1. Make a file called `app/importers/modular_importer.rb` that contains just enough of an importer class that your test can run and give a meaningful error:
+  ```ruby
+    class ModularImporter
+      def initialize(csv_file)
+        @csv_file = csv_file
+        raise "Cannot find expected input file #{csv_file}" unless File.exist?(csv_file)
+      end
+
+      def import
+      end
+    end
+  ```
+1. Run your test:
+  ```
+  bundle exec rspec spec/importers/modular_importer_spec.rb
+  ```
+  It should fail with a message like
+  ```
+  expected `Work.count` to have changed by 3, but was changed by 0
+  ```
+
+So, at this point, your test is running, but the importer isn't yet creating any records.
+
+### 2. Get the test passing
+1. Add the zizia gem to your `Gemfile` and run bundle update:
+  ```
+    gem `zizia`
+  ```
+2. Edit `app/importer/modular_importer.rb` so it looks like this:
+  ```ruby
+    require 'zizia'
+
+    class ModularImporter
+      def initialize(csv_file)
+        @csv_file = csv_file
+        raise "Cannot find expected input file #{csv_file}" unless File.exist?(csv_file)
+      end
+
+      def import
+        file = File.open(@csv_file)
+        Zizia::Importer.new(parser: Zizia::CsvParser.new(file: file), record_importer: Zizia::HyraxRecordImporter.new).import
+        file.close # Note that we must close any files we open.
+      end
+    end
+  ```
+3. Now your test should pass with output something like this:
+
+  ```
+  ModularImporter
+  Creating record: ["A Cute Dog"].Record created at: jw827b648Record created at: jw827b648Creating record: ["An Interesting Cat"].Record created at: 3n203z084Record created at: 3n203z084Creating record: ["A Flock of Birds"].Record created at: wm117n96bRecord created at: wm117n96b  imports a csv
+
+  Finished in 7.56 seconds (files took 9.06 seconds to load)
+  1 example, 0 failures
+  ```
+
+### 3. Import in development and production
+
+A passing test tell us that records are being created, but it's only happening in our test environment right now. Let's write a rake task so we can make it happen in development or production and really see the records we made.
+
+Make a file like this at `lib/tasks/import.rake`:
+
+```ruby
+  namespace :basic_import do
+    desc 'Ingest sample data'
+    task sample: [:environment] do
+      Rake::Task["hyrax:default_admin_set:create"].invoke
+      Rake::Task["hyrax:default_collection_types:create"].invoke
+      Rake::Task["hyrax:workflow:load"].invoke
+      csv_file = Rails.root.join('spec', 'fixtures', 'csv_import', 'zizia_basic.csv')
+      ModularImporter.new(csv_file).import
+    end
+  end
+```
+
+Now you should be able to run `rake basic_import:sample` and see your records appear in Hyrax.

data/lib/zizia/always_invalid_validator.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module Zizia
+  ##
+  # A Validator that always gives an error named `:everytime`.
+  #
+  # @example
+  #   validator = AlwaysInvalidValidator.new
+  #   validator.validate(:anything, :at, :all) # => [Error<#...>]
+  class AlwaysInvalidValidator < Validator
+    ##
+    # @return [Array<Validator::Error>]
+    def validate(*)
+      [Error.new(self, :everytime)]
+    end
+  end
+end

data/lib/zizia/csv_template.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module Zizia
+  class CsvTemplate
+    def to_s
+      Zizia.config.metadata_mapper_class.new.fields.join(',')
+    end
+  end
+end

data/lib/zizia/engine.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+require 'rails/all'
+require 'carrierwave'
+require 'carrierwave/orm/activerecord'
+require 'redcarpet'
+require 'devise'
+require 'hyrax'
+require 'riiif'
+require 'hydra-role-management'
+
+module Zizia
+  class Engine < ::Rails::Engine
+    isolate_namespace Zizia
+
+    config.generators do |g|
+      g.test_framework :rspec
+    end
+
+    initializer :zizia_assets_precompile do |app|
+      app.config.assets.precompile << %w[zizia/application.js zizia/application.css]
+    end
+
+    initializer :append_migrations do |app|
+      unless app.root.to_s.match root.to_s
+        config.paths['db/migrate'].expanded.each do |expanded_path|
+          app.config.paths['db/migrate'] << expanded_path
+        end
+      end
+    end
+  end
+end

data/lib/zizia/hash_mapper.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+module Zizia
+  ##
+  # A generic metadata mapper for input records
+  #
+  # Maps from hash accessor syntax (`['title']`) to method call dot syntax (`.title`)
+  #
+  # The fields provided by this mapper are dynamically determined by the fields
+  # available in the provided metadata hash.
+  #
+  # All field values are given as multi-valued arrays.
+  #
+  # @example
+  #   mapper = HashMapper.new
+  #   mapper.fields # => []
+  #
+  #   mapper.metadata = { title: 'Comet in Moominland', author: 'Tove Jansson' }
+  #   mapper.fields # => [:title, :author]
+  #   mapper.title  # => ['Comet in Moominland']
+  #   mapper.author # => ['Tove Jansson']
+  #
+  class HashMapper < MetadataMapper
+    ##
+    # @param meta [#to_h]
+    # @return [Hash<String, String>]
+    def metadata=(meta)
+      @metadata = meta.to_h
+    end
+
+    ##
+    # @return [Enumerable<Symbol>] The fields the mapper can process
+    def fields
+      return [] if metadata.nil?
+      metadata.keys.map(&:to_sym)
+    end
+
+    ##
+    # @see MetadataMapper#map_field
+    def map_field(name)
+      Array(metadata[name.to_s])
+    end
+  end
+end

data/lib/zizia/hyrax/hyrax_basic_metadata_mapper.rb

@@ -0,0 +1,149 @@
+# frozen_string_literal: true
+require 'uri'
+
+module Zizia
+  ##
+  # A mapper for Hyrax metadata.
+  #
+  # Maps from hash accessor syntax (`['title']`) to method call dot syntax (`.title`).
+  #
+  # The fields provided by this mapper are the same as the properties defined in `Hyrax::CoreMetadata` and `Hyrax::BasicMetadata`.
+  #
+  # @note This mapper allows you to set values for all the Hyrax fields, but depending on how you create the records, some of the values might get clobbered.  For example, if you use Hyrax's actor stack to create records, it might overwrite fields like `date_modified` or `depositor`.
+  #
+  # @see HashMapper Parent class for more info and examples.
+  class HyraxBasicMetadataMapper < HashMapper
+    # If your CSV headers don't exactly match the
+    # the method name for the property's setter
+    # method, add a mapping here.
+    # Example: the method name is work.resource_type,
+    # but in the CSV file, the header is
+    # 'resource type' (without the underscore).
+    CSV_HEADERS = {
+      resource_type: 'resource type',
+      description: 'abstract or summary',
+      rights_statement: 'rights statement',
+      date_created: 'date created',
+      based_near: 'location',
+      related_url: 'related url'
+    }.freeze
+
+    ##
+    # @return [Enumerable<Symbol>] The fields the mapper can process.
+    def fields
+      core_fields + basic_fields + [:visibility]
+    end
+
+    # Properties defined with `multiple: false` in
+    # Hyrax should return a single value instead of
+    # an Array of values.
+    def depositor
+      single_value('depositor')
+    end
+
+    def date_modified
+      single_value('date_modified')
+    end
+
+    def label
+      single_value('label')
+    end
+
+    def relative_path
+      single_value('relative_path')
+    end
+
+    def import_url
+      single_value('import_url')
+    end
+
+    # We should accept visibility values that match the UI and transform them into
+    # the controlled vocabulary term expected by Hyrax
+    def visibility
+      case metadata[matching_header('visibility')]&.downcase&.gsub(/\s+/, "")
+      when 'public'
+        'open'
+      when 'open'
+        'open'
+      when 'registered'
+        'authenticated'
+      when "authenticated"
+        'authenticated'
+      when ::Hyrax::Institution.name&.downcase&.gsub(/\s+/, "")
+        'authenticated'
+      when ::Hyrax::Institution.name_full&.downcase&.gsub(/\s+/, "")
+        'authenticated'
+      when 'private'
+        'restricted'
+      when 'restricted'
+        'restricted'
+      else
+        'restricted' # This is the default if nothing else matches
+      end
+    end
+
+    def files
+      map_field('files')
+    end
+
+    ##
+    # @return [String] The delimiter that will be used to split a metadata field into separate values.
+    # @example
+    #   mapper = HyraxBasicMetadataMapper.new
+    #   mapper.metadata = { 'language' => 'English|~|French|~|Japanese' }
+    #   mapper.language = ['English', 'French', 'Japanese']
+    #
+    def delimiter
+      @delimiter ||= '|~|'
+    end
+    attr_writer :delimiter
+
+    ##
+    # @see MetadataMapper#map_field
+    def map_field(name)
+      method_name = name.to_s
+      method_name = CSV_HEADERS[name] if CSV_HEADERS.keys.include?(name)
+      key = matching_header(method_name)
+      Array(metadata[key]&.split(delimiter))
+    end
+
+    protected
+
+      # Some fields should have single values instead
+      # of array values.
+      def single_value(field_name)
+        metadata[matching_header(field_name)]
+      end
+
+      # Lenient matching for headers.
+      # If the user has headers like:
+      #   'Title' or 'TITLE' or 'Title  '
+      # it should match the :title field.
+      def matching_header(field_name)
+        metadata.keys.find do |key|
+          next unless key
+          key.downcase.strip == field_name
+        end
+      end
+
+      # Properties defined in Hyrax::CoreMetadata
+      # Note that date_uploaded is NOT set here, even though it is defined in
+      # Hyrax::CoreMetadata. Hyrax expects to set date_uploaded itself, and
+      # sending a metadata value for that field interferes with Hyrax expected
+      # behavior.
+      def core_fields
+        [:depositor, :title, :date_modified]
+      end
+
+      # Properties defined in Hyrax::BasicMetadata
+      def basic_fields
+        [:label, :relative_path, :import_url,
+         :resource_type, :creator, :contributor,
+         :description, :keyword, :license,
+         :rights_statement, :publisher, :date_created,
+         :subject, :language, :identifier,
+         :based_near, :related_url,
+         :bibliographic_citation, :source]
+      end
+  end
+end

data/lib/zizia/hyrax/hyrax_default_updater.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+module Zizia
+  class HyraxDefaultUpdater < HyraxMetadataOnlyUpdater
+    attr_reader :attrs
+
+    def actor_stack
+      terminator = Hyrax::Actors::Terminator.new
+      Hyrax::DefaultMiddlewareStack.build_stack.build(terminator)
+    end
+  end
+end

data/lib/zizia/hyrax/hyrax_delete_files_updater.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+module Zizia
+  class HyraxDeleteFilesUpdater < HyraxMetadataOnlyUpdater
+    attr_reader :attrs
+
+    def actor_stack
+      terminator = Hyrax::Actors::Terminator.new
+      Hyrax::DefaultMiddlewareStack.build_stack.build(terminator)
+    end
+
+    def update
+      existing_record.file_sets.map(&:destroy)
+      super
+    end
+  end
+end

data/lib/zizia/hyrax/hyrax_metadata_only_updater.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+module Zizia
+  class HyraxMetadataOnlyUpdater
+    attr_accessor :depositor,
+                  :existing_record,
+                  :update_record,
+                  :based_near_attributes,
+                  :csv_import_detail
+
+    def initialize(csv_import_detail:,
+                   existing_record:,
+                   update_record:,
+                   attrs:)
+      @csv_import_detail = csv_import_detail
+      @depositor = ::User.find(csv_import_detail.depositor_id)
+      @update_record = update_record
+      @existing_record = existing_record
+      @attrs = attrs
+    end
+
+    def attrs
+      @attrs.reject { |k, _v| k == :uploaded_files }
+    end
+
+    def actor_stack
+      terminator = Hyrax::Actors::Terminator.new
+      Zizia::MetadataOnlyStack.build_stack.build(terminator)
+    end
+
+    def started
+      Rails.logger.info "[zizia] event: record_update_started, batch_id: #{csv_import_detail.batch_id}, collection_id: #{csv_import_detail.collection_id}, #{csv_import_detail.deduplication_field}: #{update_record.respond_to?(csv_import_detail.deduplication_field) ? update_record.send(csv_import_detail.deduplication_field.deduplication_field) : update_record}"
+    end
+
+    def succeeded
+      Rails.logger.info "[zizia] event: record_updated, batch_id: #{csv_import_detail.batch_id}, record_id: #{csv_import_detail.existing_record.id}, collection_id: #{csv_import_detail.collection_id}, #{csv_import_detail.deduplication_field}: #{existing_record.respond_to?(csv_import_detail.deduplication_field) ? existing_record.send(csv_import_detail.deduplication_field) : existing_record}"
+    end
+
+    def failed(attr)
+      Rails.logger.error "[zizia] event: validation_failed, batch_id: #{csv_import_detail.batch_id}, collection_id: #{csv_import_detail.collection_id}, attribute: #{attr.capitalize}, message: #{msg}, record_title: record_title: #{attrs[:title] ? attrs[:title] : attrs}"
+    end
+
+    def create_actor_env
+      Hyrax::Actors::Environment.new(existing_record, ::Ability.new(depositor), attrs)
+    end
+
+    def update
+      if actor_stack.update(create_actor_env)
+        csv_import_detail.success_count += 1
+      else
+        existing_record.errors.each_key do |attr, _msg|
+          failed(attr)
+        end
+        csv_import_detail.failure_count += 1
+      end
+      csv_import_detail.save
+    end
+  end
+end