mets_converter 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: c5dc746835db25047ca43467efca3ea3be74ba5ab281bd0d2bb89aef305a29da
+  data.tar.gz: 9fe15a4c78a4ac5b503b8e6f9879ec3860fbace9b67ac2a67006107a41a689e7
+SHA512:
+  metadata.gz: 7d99159d56af0265d969230fb63f99a17abce666396f6299e60d5afb602ab8cfae7f5514242dffd14d968cb385f24762f953b5586641b7481f32060469eb3888
+  data.tar.gz: e571d6d8e43c96d00be1b1d5f4f1ba8427b21ceb4a4636df087797f5f6966664cfd1c88a08ead5fd0bff9457b43004fb231a19e6e977f6b95d336637b5c4abf0

data/.gitignore

@@ -0,0 +1,13 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+tags
+
+# rspec failure tracking
+.rspec_status

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.6.3
+before_install: gem install bundler -v 2.0.2

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at brendan-quinn@northwestern.edu. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in mets_converter.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,15 @@
+##########################################################################
+# Copyright 2019 Northwestern University Library
+# Additional copyright may be held by others, as reflected in the commit log
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

data/README.md

@@ -0,0 +1,49 @@
+# MetsConverter
+
+[![Build Status](https://travis-ci.com/nulib/mets_converter.svg)](https://travis-ci.com/nulib/mets_converter)
+
+Convert a mets xml file into a yml file according to the Hathi Trust specifications for ingest. Note: this project was developed to satisfy Northwestern Libraries' specific use case, so many values are hard-coded in `MetsConverter::YamlBuilder#build`.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'mets_converter'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install mets_converter
+
+Once installed, the `mets_to_yaml` command line utility should be available in your terminal.
+
+## Usage
+
+Run `mets_to_yaml` without any arguments to see help text.
+
+```sh
+# Running the mets_to_yaml command
+mets_to_yaml [options] input_file output_file_basename
+```
+
+```sh
+# Example with optional arguments
+mets_to_yaml --force --resolution="300" /cygdrive/e/books/limb_output/35556004429411/35556004429411.mets.xml meta
+
+# meta.yml will be created in the same directory as the input file
+```
+
+## Development
+
+After checking out the repo, run `bundle install` to install dependencies. Then, run `bundle exec rspec` to run the tests.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/nulib/mets_converter. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.

data/Rakefile

@@ -0,0 +1,6 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'mets_converter'
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require 'irb'
+IRB.start(__FILE__)

data/bin/mets_to_yaml

@@ -0,0 +1,84 @@
+#!/usr/bin/env ruby
+
+require 'mets_converter'
+require 'optparse'
+
+options = {}
+
+option_parser = OptionParser.new do |opts| # rubocop:disable Metrics/BlockLength
+  executable_name = File.basename($PROGRAM_NAME)
+  opts.banner = "Usage: #{executable_name} [options] input_file output_file_name"
+
+  opts.banner = <<-EOS
+  Convert a mets xml file into a yml file according to the Hathi Trust specifications for ingest
+  Usage: #{executable_name} [options] input_file output_file_basename
+  Example usage: #{executable_name} --force --resolution="300" /cygdrive/e/books/limb_output/35556004429411/35556004429411.mets.xml meta
+  Example output: meta.yml
+  Options:
+  EOS
+
+  opts.on('--[no-]force',
+          'Overwrite existing files') do |force| # (1)
+    options[:force] = force
+  end
+
+  opts.on('--suprascan',
+          '"SupraScan Quartz A1" will override default "Kirtas APT 1200"') do
+    options[:suprascan] = true
+  end
+
+  opts.on('--resolution=resolution_integer',
+          'Enter numeric value of scanning resolution to override default 300') do |resolution|
+    options[:resolution] = resolution
+  end
+
+  opts.on('--scanning_order_rtl',
+          'Use if the scanning order is right-to-left to override default left-to-right') do
+    options[:scanning_order_rtl] = true
+  end
+
+  opts.on('--reading_order_rtl',
+          'Use if the reading order is right-to-left to override default left-to-right') do
+    options[:reading_order_rtl] = true
+  end
+end
+
+begin
+  $stdout.puts 'Parsing optional arguments...'
+  option_parser.parse!
+  if ARGV.length < 2
+    puts 'error: you must supply an input file and an output file name'
+    puts option_parser.help
+    exit 2
+  end
+rescue OptionParser::InvalidArgument => ex
+  $stderr.puts ex.message
+  $stderr.puts option_parser
+  exit 1
+end
+
+# set output file based on second command-line argument
+output_file_basename = ARGV[1]
+$stdout.puts "Building #{output_file_basename}.yml..."
+
+# create mets object from first command-line argument
+mets = MetsConverter::MetsParser.new(ARGV[0])
+raise "The file #{ARGV[0]} does not contain valid XML" if mets.document.nil?
+
+# build yaml combining mets data and command-line options
+yaml = MetsConverter::YamlBuilder.new(mets, options).build
+output_file = File.join(mets.input_file_location, "#{output_file_basename}.yml")
+
+if File.exist? output_file
+  if options[:force]
+    $stdout.puts "Overwriting #{output_file}"
+  else
+    $stderr.puts "#{output_file} already exists, use --force to overwrite"
+    exit 1
+  end
+end
+
+unless ENV['NO_RUN']
+  File.write(output_file, yaml)
+  $stdout.puts "Created #{output_file} successfully"
+end

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/indent.rb

@@ -0,0 +1,5 @@
+String.class_eval do
+  def indent(num_spaces)
+    sub(/^/, ' ' * num_spaces)
+  end
+end

data/lib/mets_converter/logging.rb

@@ -0,0 +1,26 @@
+require 'logger'
+
+module MetsConverter
+  module Logging
+    def self.initialize_logger(log_target = STDOUT)
+      oldlogger = defined?(@logger) ? @logger : nil
+      @logger = Logger.new(log_target)
+      @logger.level = Logger::INFO
+      @logger.datetime_format = '%Y-%m-%d %H:%M:%S '
+      oldlogger.close if oldlogger
+      @logger
+    end
+
+    def self.logger
+      defined?(@logger) ? @logger : initialize_logger
+    end
+
+    def self.logger=(log)
+      @logger = (log ? log : MetsConverter::Logging.logger.new(File::NULL))
+    end
+
+    def logger
+      MetsConverter::Logging.logger
+    end
+  end
+end

data/lib/mets_converter/mets_parser.rb

@@ -0,0 +1,37 @@
+require 'nokogiri'
+
+module MetsConverter
+  class MetsParser
+    attr_reader :document
+
+    def initialize(file)
+      @file = file
+      @document = parse_xml
+    end
+
+    def input_file_location
+      File.dirname(@file)
+    end
+
+    def capture_date
+      document.xpath('//xmlns:metsHdr')[0].attr('CREATEDATE') + '-06:00'
+    end
+
+    def pages
+      document.search('structMap[@TYPE="logical"]//div[@TYPE="page"]')
+    end
+
+    private
+
+    def parse_xml
+      input_file = File.open(@file)
+      begin
+        xsd = Nokogiri::XML::Schema(File.open('xsd/mets.xsd'))
+        doc = Nokogiri::XML(input_file, &:strict)
+        xsd.valid?(doc) ? doc : raise('Invalid METS file')
+      rescue Nokogiri::XML::SyntaxError => e
+        MetsConverter.logger.error "XML error: #{e}"
+      end
+    end
+  end
+end

data/lib/mets_converter/version.rb

@@ -0,0 +1,3 @@
+module MetsConverter
+  VERSION = '0.2.0'.freeze
+end

data/lib/mets_converter/yaml_builder.rb

@@ -0,0 +1,147 @@
+require_relative '../indent'
+require 'pry'
+
+module MetsConverter
+  class YamlBuilder
+
+    attr_reader :document, :pages, :options
+
+    def initialize(mets, **options)
+      @document = mets.document
+      @pages = mets.pages
+      @options = options
+    end
+
+    def build
+      # Descriptive and technical information
+
+      # Capture Date
+      #   <metsHdr CREATEDATE="2015-07-01T15:26:39" RECORDSTATUS="Complete">
+      yaml = "capture_date: #{document.xpath("//xmlns:metsHdr")[0].attr("CREATEDATE")}-06:00\n"
+      # Scanner Make and Model
+      if options[:suprascan]
+        yaml += "scanner_make: SupraScan\n"
+        yaml += "scanner_model: Quartz A1\n"
+      else
+        yaml += "scanner_make: Kirtas\n"
+        yaml += "scanner_model: APT 1200\n"
+      end
+      # Scanner User
+      yaml += "scanner_user: \"Northwestern University Library: Repository & Digital Curation\"\n"
+      # Resolution
+      yaml += "contone_resolution_dpi: #{options[:resolution] || 300}\n"
+      # Image Compression Date
+      yaml += "image_compression_date: #{document.xpath("//xmlns:metsHdr")[0].attr("CREATEDATE")}-06:00\n"
+      # Image Compression Agent
+      yaml += "image_compression_agent: northwestern\n"
+      # Image Compression Tool
+      yaml += "image_compression_tool: [\"LIMB v3.1.0.0\"]\n"
+      # Scanning Order
+      if options[:scanning_order_rtl]
+        yaml += "scanning_order: right-to-left\n"
+      else
+        yaml += "scanning_order: left-to-right\n"
+      end
+      # Reading Order
+      if options[:reading_order_rtl]
+        yaml += "reading_order: right-to-left\n"
+      else
+        yaml += "reading_order: left-to-right\n"
+        yaml += "pagedata:\n"
+      end
+
+      # File List
+
+      # Loop through pages within logical structMap
+      pages.each do |element|
+        # Store the fileid for the jp2
+        file_id = element.xpath('./xmlns:fptr[starts-with(@FILEID, "JP2")]')[0]["FILEID"]
+        # Store the jp2 filename
+        filename = find_filename_by_file_id(file_id)
+        # Since the yaml flattens out the xml structure,
+        # the first child of each parent gets special treatment (of course)
+        # i.e. labels for covers, titles, chapters, etc.
+        if element == element.parent.first_element_child
+          case
+          when element.parent["LABEL"] == "Cover" && element.parent["TYPE"] == "cover" && element.parent == document.search('structMap[@TYPE="logical"]//div[@TYPE="cover"]').first
+            if element["ORDERLABEL"].empty?
+              line = filename + ": { label: \"FRONT_COVER\" }\n"
+            else
+              line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\", label: \"FRONT_COVER\" }\n"
+            end
+          when element.parent["LABEL"] == "Front Matter"
+            next if element["ORDERLABEL"].empty?
+            line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\" }\n"
+          when element.parent["LABEL"] == "Cover" && element.parent["TYPE"] == "appendix"
+            next if element["ORDERLABEL"].empty?
+            line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\" }\n"
+          when element.parent["LABEL"] == "Title"
+            if element["ORDERLABEL"].empty?
+              line = filename + ": { label: \"TITLE\" }\n"
+            else
+              line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\", label: \"TITLE\" }\n"
+            end
+          when element.parent["LABEL"] == "Contents"
+            if element["ORDERLABEL"].empty?
+              line = filename + ": { label: \"TABLE_OF_CONTENTS\" }\n"
+            else
+              line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\", label: \"TABLE_OF_CONTENTS\" }\n"
+            end
+          when element.parent["LABEL"] == "Preface"
+            if element["ORDERLABEL"].empty?
+              line = filename + ": { label: \"PREFACE\" }\n"
+            else
+              line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\", label: \"PREFACE\" }\n" 
+            end
+          # First page within the body, can be within a div with label attribute "Introduction" or "Chapter"
+          when element == document.at('structMap[@TYPE="logical"]//div[@TYPE="body"]/div[1]/div[1]') && (element.parent["LABEL"] == "Introduction" || element.parent["LABEL"].start_with?("Chapter"))
+            if element["ORDERLABEL"].empty?
+              line = filename + ": { label: \"FIRST_CONTENT_CHAPTER_START\" }\n"
+            else
+              line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\", label: \"FIRST_CONTENT_CHAPTER_START\" }\n"
+            end
+          when element.parent["LABEL"] == "Back Matter"
+            next if element["ORDERLABEL"].empty?
+            line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\" }\n"
+          when element.parent["LABEL"].start_with?("Chapter") || element.parent["LABEL"] == "Appendix"
+            if element["ORDERLABEL"].empty?
+              line = filename + ": { label: \"CHAPTER_START\" }\n"
+            else
+              line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\", label: \"CHAPTER_START\" }\n"
+            end
+          when element.parent["LABEL"] == "Notes" || element.parent["LABEL"] == "Bibliography"
+            if element["ORDERLABEL"].empty?
+              line = filename + ": { label: \"REFERENCES\" }\n"
+            else
+              line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\", label: \"REFERENCES\" }\n"
+            end
+          when element.parent["LABEL"] == "Index"
+            if element["ORDERLABEL"].empty?
+              line = filename + ": { label: \"INDEX\" }\n"
+            else
+              line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\", label: \"INDEX\" }\n"
+            end
+          when element.parent["LABEL"] == "Cover" && element.parent["TYPE"] == "cover" && element.parent == document.search('structMap[@TYPE="logical"]//div[@TYPE="cover"]').last
+            if element["ORDERLABEL"].empty?
+              line = filename + ": { label: \"BACK_COVER\" }\n"
+            else
+              line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\", label: \"BACK_COVER\" }\n"
+            end
+          end
+        else
+          # remaining pages
+          # skip pages that don't have page numbers (stored in "ORDERLABEL" attribute)
+          next if element["ORDERLABEL"].empty?
+          line = filename + ": { orderlabel: \"#{element["ORDERLABEL"]}\" }\n"
+        end
+        yaml += line.indent(4) if line
+      end
+
+      yaml
+    end
+
+    def find_filename_by_file_id(id)
+      document.xpath("//xmlns:file[@ID=\"#{id}\"]/xmlns:FLocat")[0]['xlink:href'][7..-1]
+    end
+  end
+end

data/lib/mets_converter.rb

@@ -0,0 +1,13 @@
+require 'mets_converter/mets_parser'
+require 'mets_converter/version'
+require 'mets_converter/yaml_builder'
+
+module MetsConverter #:nodoc:
+  def self.logger
+    MetsConverter::Logging.logger
+  end
+
+  def self.logger=(log)
+    MetsConverter::Logging.logger = log
+  end
+end

data/mets_converter.gemspec

@@ -0,0 +1,27 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'mets_converter/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'mets_converter'
+  spec.version       = MetsConverter::VERSION
+  spec.authors       = ['Brendan Quinn']
+  spec.email         = ['brendan-quinn@northwestern.edu']
+
+  spec.summary       = 'Converts METS schema encoded XML to YML'
+  spec.description   = 'Converts METS schema encoded XML to YML'
+  spec.homepage      = 'https://github.com/nulib/mets_converter'
+
+  spec.files         = `git ls-files`.split($/)
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.executables   = ["mets_to_yaml"]
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'nokogiri', '>= 1.10.4'
+
+  spec.add_development_dependency 'bundler', '~> 2.0'
+  spec.add_development_dependency 'pry'
+  spec.add_development_dependency 'rake', '~> 13.0'
+  spec.add_development_dependency 'rspec', '~> 3.9'
+end