cdmdexer 0.17.7

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8d3dccb39ef4048b79af386f1d3696f9d7497efb3c53e388480e1505e027d99b
+  data.tar.gz: 262051a8e4e246be6092a5cbc54245cbed91449c859f5c408bde2f7b8d9ae068
+SHA512:
+  metadata.gz: 41aeca2b754fc5681e5bfe120e4d690e4f56f5b767bd924f50a0363ca2fdfad2238e2c58507d61e79805c37e0218267903e662c968768c5d20fce807d98cecc9
+  data.tar.gz: 2a99d19a2639e15b5811692101d9630a67296b9ee1a32bda4e1cf70be5dcf4a0aba8815602b9172b935b8bf1482531026f5ebc8ee5289439a4baee76a7b6873c

data/.gitignore

@@ -0,0 +1,10 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.gem

data/.rubocop.yml

@@ -0,0 +1,4 @@
+Layout/IndentationWidth:
+  # Number of spaces for each indentation level.
+  Width: 2
+  IgnoredPatterns: []

data/.travis.yml

@@ -0,0 +1,8 @@
+sudo: false
+language: ruby
+rvm:
+- 2.3.0
+before_install: gem install bundler -v 1.12.5
+env:
+  matrix:
+    secure: n+qvS/V5ZeCZ15RMNPeTxtAd+79XZniWNoYCriGtrJkELt/7bc4teXB8thowXHrm4Sx2lHW9uE8cR2s56RyRoV0PQU+nenjylAHPkfwlu/isutw3PGGFJoF3O89bG5jtQCQcequSH+PHRLc0xYoKaRLOMEuO0RPzd+hALt3jobzs5Q4hvvFinF8yDOTG0Qo24CtvZqNMAz0QtkWJExIZ8i2G7DnkY7Jy6DaJjv+A3MdTe2xTikSjWOvFjMrRVd1v+aP9gUTNVTy5aahpKSwub7TzsS/yj8rSI8717As/sSxepWXBg1N1AnrH6ttgjpdyjELXYlTrCx01u4sQMW3HRKG4WJzSVKaQ9W6fF82TQySh6hAaiO5/HYZ73+TTCnEk6eFJT/FaT7yNyOwhRA2Wq091PVFgfZOFy+Hxbj3dWVZm0hS5odaQ5c/aUYR8EiyfUwgHQQxl2QYjUMci+5QOa5k5TmqBHq4iqa0khC5t/zbCvY5ImC4/ySTtIQZnAtHxh3ZhdrukQn0LgLQ4BKVGZEz0wyiaMQrkf8XUgv5pkegTcAhkaz7vZs5j0wtY6CV0q9t5D2n8Xj9amo3SOfTtJjOLZ3xBJhjJRr4J219wN0EFtHt+3rXVoKcw3Vz1ysyV1Jn2B68LF0LPHwSFKDGqptP3axBJNs/utGcDTHAx8hU=

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,49 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at fenne035@umn.edu. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in cdmdexer.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 University of Minnesota Libraries
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,149 @@
+[![Build Status](https://travis-ci.org/UMNLibraries/cdmdexer.svg?branch=master)](https://travis-ci.org/UMNLibraries/cdmdexer)
+
+# CDMDEXER: Index CONTENTdm Content
+
+A micro [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load) system dedicated to extracting metadata records from a CONTENTdm instance (using the [CONTENTdm API gem](https://github.com/UMNLibraries/contentdm_api), transforming them into Solr documents, and loading them into Solr.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'cdmdexer'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install cdmdexer
+
+Add the CDMDEXER rake task to your project Rakefile:
+
+```ruby
+require 'cdmdexer/rake_task'
+```
+
+### GeoNames (optional)
+
+In order to make use of the GeoNames service, you must purchase a [GeoNames Premium Webservices Account](http://www.geonames.org/commercial-webservices.html). If you do not have a `geonam` field in your CONTENTdm schema, you may ignore this instruction. Add your credentials to your shell environment once you have secured a GeoNames user:
+
+```
+# e.g. within your .bash_profile or .zprofile file
+export export GEONAMES_USER="yourusernamehere"
+```
+
+## Usage
+
+Run the ingester
+
+rake cdmdexer:batch[solr_url,oai_endpoint,cdm_endpoint,set_spec, batch_size, max_compounds]
+
+|Argument| Definition|
+|--:|---|
+|solr_url| The full URL to your Solr core instance (same as your blacklight.yml solr url)|
+|oai_endpoint| A URL to your OAI instance (e.g. https://server16022.contentdm.oclc.org/oai/oai.php)   |
+|cdm_endpoint| A URL to your CONTENTdm API endpoint (e.g. https://server16022.contentdm.oclc.org/dmwebservices/index.php) |
+|set_spec| Selectively harvest from a single collection with [setSpec](http://www.openarchives.org/OAI/openarchivesprotocol.html#Set)|
+|batch_size| The number of records to transform at a time. **Note**: it is within the record transformation process that the CONTENTdm API is requested. This API can be sluggish, so we conservatively transform batches of ten records at a time to prevent timeouts.|
+|max_compounds| CONTENTdm records with many compounds can take a long time to load from the CONTENTdm API as multiple requests must happen in order to get the metadata for each child record of a parent compound object. For this reason, records with ten or more compound children are, by default, processed in batches of one. This setting allows you to override this behavior.|
+
+For example:
+
+```ruby
+rake "cdmdexer:ingest[http://solr:8983/solr/foo-bar-core, https://server16022.contentdm.oclc.org/oai/oai.php, https://server16022.contentdm.oclc.org/dmwebservices/index.php, 2015-01-01]"
+```
+
+### Custom Rake Tasks
+
+You might also create your own rake task to run your modified field transformers:
+
+```ruby
+require 'cdmdexer'
+
+namespace :cdmdexer do
+  desc "ingest batches of records"
+  ##
+  # e.g. rake mdl_ingester:ingest[2015-09-14, 2]
+  task :batch, [:batch_size, :set_spec] => :environment  do |t, args|
+    config  =
+      {
+        oai_endpoint: 'http://cdm16022.contentdm.oclc.org/oai/oai.php',
+        cdm_endpoint: 'https://server16022.contentdm.oclc.org/dmwebservices/index.php',
+        set_spec: (args[:set_spec] != '""') ? args[:set_spec] : nil,
+        batch_size: (args[:batch_size]) ? args[:batch_size] : 30,
+        solr_config: solr_config
+      }
+    CDMDEXER::ETLWorker.perform_async(config)
+  end
+end
+```
+### Your Own Custom Solr Field Mappings (see above code snippet)
+
+The default CONTENTdm to Solr field transformation rules may be overriden by calling the CDMDEXER::ETLWorker (a [Sidekiq worker](https://github.com/mperham/sidekiq)) directly. These rules may be found in the default_mappings method of the [CDMDEXER::Transformer Class](https://github.com/UMNLibraries/cdmdexer/blob/master/lib/cdmdexer/transformer.rb).
+
+The transformer expects mappings in the following format:
+
+```ruby
+def your_custom_field_mappings
+  [
+    {dest_path: 'title_tei', origin_path: 'title', formatters: [StripFormatter]},
+  ]
+end
+```
+|Argument| Definition|
+|--:|---|
+|dest_path| The 'destination path' is the name of the field you will be sending to Solr for this field mapping. |
+|origin_path| Where to get the field data from the original record for this mapping. |
+|formatters| [Formatters](https://github.com/UMNLibraries/cdmdexer/blob/master/lib/cdmdexer/formatters.rb) perform tasks such as stripping white space or splitting CONTENTdm multi-valued fields (delimited by semicolons) into JSON arrays. |
+
+**Note:** The first formatter receives the value found at the declared `origin_path`. Each formatter declared after the initial formatter will receive a value produced by the preceding formatter.
+
+Formatters are very simple stateless classes that take a value, do something to it, and respond with a modified version of this value via a class method called `format`. Examples of other formatters may be found in the [Formatters file](https://github.com/UMNLibraries/cdmdexer/blob/master/lib/cdmdexer/formatters.rb). For Example:
+
+```ruby
+  class SplitFormatter
+    def self.format(value)
+      (value.respond_to?(:split)) ? value.split(';') : value
+    end
+  end
+```
+
+You might also want to simply override some of the default mappings or add your own:
+
+```ruby
+mappings = CDMDEXER::Transformer.default_mappings.merge(your_custom_field_mappings)
+```
+## A Custom Post-indexing Callback
+
+If you would like to perform some action (e.g. send an email) following the completion of the CDMDEXER indexing process, you may declare your own callback hook (anything with "Callback" in the class name declared within the CDMDEXER module space will be used). To do so in Rails, create a Rails initializer file `config/initializers/cdmdexer.rb`:
+
+```ruby
+module CDMDEXER
+  class Callback
+    def self.call!
+      Rails.logger.info("My Custom CDMDEXER Callback")
+    end
+  end
+end
+```
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/UMNLibraries/cdmdexer. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## License
+
+[MIT](/LICENSE.txt)
+
+## TODO
+
+* Make StripFormatter the default formatter so it doesn't need to be declared for every field
+* Re-brand project: CONTENTdm Indexer. CDMDEXER doesn't necessarily require Blacklight. Moreover only handles indexing.

data/Rakefile

@@ -0,0 +1,11 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+require 'cdmdexer/rake_task'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList['test/**/*_test.rb']
+end
+
+task :default => :test

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "cdmdexer"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/cdmdexer.gemspec

@@ -0,0 +1,35 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'cdmdexer/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'cdmdexer'
+  spec.version       = CDMDEXER::VERSION
+  spec.authors       = ['chadfennell']
+  spec.email         = ['fenne035@umn.edu']
+
+  spec.summary       = %q{Load CONTENTdm data into a Solr Index. CDMDEXER expects to run inside a Rails application.}
+  spec.license       = 'MIT'
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = 'exe'
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+
+  spec.add_dependency 'hash_at_path', '~> 0.1'
+  spec.add_dependency 'contentdm_api', '~> 0.5.0'
+  spec.add_dependency 'sidekiq', '>= 3.5'
+  spec.add_dependency 'titleize', '~> 1.4'
+  spec.add_dependency 'rsolr', '~> 2.0'
+  # CDMDEXER expects to run in a rails app, but just to avoid adding
+  # another external dependency for XML procssing, we rely on activesupport's
+  # Has.to_jsonl feature for testing and to allow this gem to function
+  # independently from a rails app
+  spec.add_dependency 'rails', '>= 5.2'
+
+  spec.add_development_dependency 'bundler', '~> 1.12'
+  spec.add_development_dependency 'rake', '~> 12.0'
+  spec.add_development_dependency 'minitest', '~> 5.0'
+  spec.add_development_dependency 'yard', '~> 0.9.0'
+end

data/lib/cdmdexer/cdm_item.rb

@@ -0,0 +1,89 @@
+module CDMDEXER
+  class CdmItem
+    attr_reader :cdm_endpoint,
+                :record,
+                :collection,
+                :id,
+                :cdm_api_klass,
+                :cdm_notification_klass
+
+    def initialize(record: :MISSING_RECORD,
+                   cdm_endpoint: :MISSING_ENDPOINT,
+                   cdm_api_klass: CONTENTdmAPI::Item,
+                   cdm_notification_klass: CDMDEXER::CdmNotification)
+      @record                 = record
+      @collection, @id        = record['id'].split(':')
+      @cdm_endpoint           = cdm_endpoint
+      @cdm_api_klass          = cdm_api_klass
+      @cdm_notification_klass = cdm_notification_klass
+    end
+
+    def to_h
+      # Preserve the record hash. It may contain compound data that has been
+      # resubmitted here by the transformer_worker as it recurses through
+      # compounds in order to extract their full metadata
+      @to_h ||= record.merge(metadata)
+    end
+
+    def page
+      primary_record.fetch('page', [])
+                    .each_with_index.map { |page, i| to_compound(page, i) }
+    end
+
+    private
+
+    def metadata
+      if first_page_id
+        # There are cases when we will not want to have to query for the
+        #  metadata of the first item of a compound. So, include the metadata of
+        # the first page in its parent record metadata.
+        #
+        # Use-case: you want to grab a thumbnail for the compound record. In
+        # this case, you'll need the format field of the first record in order
+        # to determine which thumbnail generation mechanism to use (e.g. CDM
+        # thumb vs getting a thumbnail for a video from Kaltura)
+        primary_record.merge('first_page' => request(first_page_id))
+      else
+        primary_record
+      end.merge(
+        'page' => page,
+        # When an item has pages, these pages are resubmitted to CdmItem
+        # as records in order to get their full metadata. But we want to
+        # remember that they are actually secondary / child pages
+        'record_type' => record.fetch('record_type', 'primary')
+      )
+    end
+
+    def first_page_id
+      (page.first || {}).fetch('id', '').split(':').last
+    end
+
+    def to_compound(page, i)
+      # raise "#{collection}:#{page['pageptr']}".inspect
+      page.merge(
+        # Child id is a combo of the page id and parent collection
+        'id' => "#{collection}:#{page['pageptr']}",
+        'parent_id' => record['id'],
+        'record_type' => 'secondary',
+        'child_index' => i
+      )
+    end
+
+    def primary_record
+      @primary_record ||= request(id)
+    end
+
+    # CDM's id format is collection/id. We use collection:id
+    def to_solr_id(record)
+      record.merge('id' => record['id'].split('/').join(':'))
+    end
+
+    def request(id)
+      cdm_notification_klass.call!(collection, id, cdm_endpoint)
+      to_solr_id(cdm_api_klass.new(base_url: cdm_endpoint,
+                        collection: collection,
+                        with_compound: false,
+                        id: id).metadata)
+    end
+  end
+end

data/lib/cdmdexer/default_cdm_notification.rb

@@ -0,0 +1,8 @@
+module CDMDEXER
+  # An example callback
+  class DefaultCdmNotification
+    def self.call!(collection, id, endpoint)
+      puts "A CONTENTdm API request notification: #{endpoint} #{collection}:#{id}"
+    end
+  end
+end

data/lib/cdmdexer/default_completed_callback.rb

@@ -0,0 +1,8 @@
+module CDMDEXER
+  # An example callback
+  class DefaultCompletedCallback
+    def self.call!(solr_client)
+      puts "A callback task"
+    end
+  end
+end

data/lib/cdmdexer/default_loader_notification.rb

@@ -0,0 +1,8 @@
+module CDMDEXER
+  # An example callback
+  class DefaultLoaderNotification
+    def self.call!(ingestables, deletables)
+      puts "A CONTENTdm API request notification: Loading #{ingestables.length} records and deleting #{deletables.length}"
+    end
+  end
+end

data/lib/cdmdexer/default_oai_notification.rb

@@ -0,0 +1,8 @@
+module CDMDEXER
+  # An example callback
+  class DefaultOaiNotification
+    def self.call!(location)
+      puts "An OAI callback task for #{location}"
+    end
+  end
+end

data/lib/cdmdexer/default_solr.rb

@@ -0,0 +1,35 @@
+require 'rsolr'
+
+module CDMDEXER
+  # Commnicate with Solr: add / delete stuff
+  class DefaultSolr
+    attr_reader :url, :client
+    def initialize(url: 'http://localhost:8983/solr/core-here', client: RSolr)
+      @url    = url
+      @client = client
+    end
+
+    def ids(start: 0)
+      connection.get('select',
+        :params => { :q => '*:*',
+          :defType => 'edismax',
+          :fl => '',
+          :rows => 10,
+          :start => start
+        }
+      )
+    end
+
+    def connection
+      @connection ||= client.connect url: url
+    end
+
+    def add(records)
+      connection.add records
+    end
+
+    def delete(ids)
+      connection.delete_by_id ids
+    end
+  end
+end

data/lib/cdmdexer/etl_by_set_specs.rb

@@ -0,0 +1,18 @@
+module CDMDEXER
+  class ETLBySetSpecs
+    attr_reader :set_specs, :etl_config, :etl_worker_klass
+    def initialize(set_specs: [:missing_setspec],
+                   etl_config: :missing_etl_config,
+                   etl_worker_klass: ETLWorker)
+      @set_specs        = set_specs
+      @etl_config       = etl_config
+      @etl_worker_klass = etl_worker_klass
+    end
+
+    def run!
+      set_specs.map do |set_spec|
+        etl_worker_klass.perform_async(etl_config.merge(set_spec: set_spec))
+      end
+    end
+  end
+end

data/lib/cdmdexer/etl_worker.rb

@@ -0,0 +1,113 @@
+require 'sidekiq'
+module CDMDEXER
+  # Extract records from OAI, delete records marked for deletion
+  # and send everything else to a transformation / load worker
+  class ETLWorker
+    include Sidekiq::Worker
+
+    extend ::Forwardable
+    def_delegators :@oai_request,
+                   :deletable_ids,
+                   :updatables,
+                   :next_resumption_token
+
+    attr_reader :config,
+                :solr_config,
+                :cdm_endpoint,
+                :oai_endpoint,
+                :field_mappings,
+                :resumption_token,
+                :batch_size,
+                :is_recursive
+
+    attr_writer :oai_request_klass,
+                :etl_worker_klass,
+                :load_worker_klass,
+                :completed_callback_klass,
+                :transform_worker_klass
+
+    def perform(config)
+      # Sidekiq stores params in JSON, so we can't inject dependencies. This
+      # results in the long set of arguments that follows. Otherwise, we'd
+      # simply inject the OAI request and extractor objects
+      @config            = config
+      @solr_config       = config.fetch('solr_config').symbolize_keys
+      @cdm_endpoint      = config.fetch('cdm_endpoint')
+      @oai_endpoint      = config.fetch('oai_endpoint')
+      @field_mappings    = config.fetch('field_mappings', false)
+      @resumption_token  = config.fetch('resumption_token', nil)
+      @batch_size        = config.fetch('batch_size', 5).to_i
+      @is_recursive      = config.fetch('is_recursive', true)
+
+      @oai_request = oai_request_klass.new(
+        endpoint_url: oai_endpoint,
+        resumption_token: resumption_token,
+        set_spec: config.fetch('set_spec', nil)
+      )
+
+      run_batch!
+      run_next_batch!
+    end
+
+    # Because Sidekiq serializes params to JSON, we provide custom setters
+    # for dependencies (normally these would be default params in the
+    # constructor) so that they may be mocked and tested
+    def completed_callback_klass
+      @completed_callback_klass ||= CDMDEXER::CompletedCallback
+    end
+
+    def etl_worker_klass
+      @etl_worker_klass ||= ETLWorker
+    end
+
+    def oai_request_klass
+      @oai_request_klass ||= OaiRequest
+    end
+
+    def load_worker_klass
+      @load_worker_klass ||= LoadWorker
+    end
+
+    def transform_worker_klass
+      @transform_worker_klass ||= TransformWorker
+    end
+
+    # Recurse through OAI batches one at a time
+    def run_next_batch!
+      if next_resumption_token && is_recursive
+        etl_worker_klass.perform_async(next_config)
+      else
+        completed_callback_klass.call!(solr_config)
+      end
+    end
+
+    private
+
+    # Extract an oai response, delete the deletables, transform and load the
+    # updatable items
+    def run_batch!
+      # Delete records that OAI has marked for deletion
+      delete_deletables!
+      transform_and_load!
+    end
+
+    def next_config
+      config.merge(resumption_token: next_resumption_token)
+    end
+
+    def transform_and_load!
+      updatables.each_slice(batch_size) do |records|
+        transform_worker_klass.perform_async(records,
+                                            solr_config,
+                                            cdm_endpoint,
+                                            oai_endpoint,
+                                            field_mappings,
+                                            batch_size)
+      end
+    end
+
+    def delete_deletables!
+      load_worker_klass.perform_async([], deletable_ids, solr_config)
+    end
+  end
+end

data/lib/cdmdexer/field_formatter.rb

@@ -0,0 +1,13 @@
+module CDMDEXER
+  class FieldFormatter
+    attr_reader :value, :formatters
+    def initialize(value: {}, formatters: [DefaultFormatter])
+      @value      = value
+      @formatters = formatters
+    end
+
+    def format!
+      formatters.reduce(value) { |memo, formatter| formatter.format(memo) }
+    end
+  end
+end