preservation 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: dfcf307b70473079a60f2801c6bd11e4f4c289d8
+  data.tar.gz: 0e13efac8904ccd96fc690644520e20f388a4fce
+SHA512:
+  metadata.gz: 939f44e9a24177232e900953f4b59b64cfe62e0bce056765444dd9a726b64b441cae7b8fc6b3cea233ef9febd732a9a27f0759992cafe4e8494213edc947fffa
+  data.tar.gz: 2a86f065c50cf43c8ea7bcaa707d278f05b7b73b2be53cbebad47e50f38ffa3e8324dc88a71ded669495e6c7f719ca7a7dbd8943da3bcf43a186f6f56df40daf

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+.idea

data/CHANGELOG.md

@@ -0,0 +1,11 @@
+# Change Log
+All notable changes to this project will be documented in this file.
+This project adheres to [Semantic Versioning](http://semver.org/).
+
+## Unreleased
+
+## 0.1.0 - 2016-09-13
+### Added
+- Transfer preparation.
+- Reporting from transfers database.
+- Disk space management.

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in preservation.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2016 Adrian Albin-Clark
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/PITCHME.md

@@ -0,0 +1,126 @@
+#HSLIDE
+
+## Rationale
+Archivematica's [Automation Tools](https://github.com/artefactual/automation-tools)
+work with files and descriptive metadata which must be provided in a certain way.
+
+
+#HSLIDE
+
+## Preservation: a way to manage ingest
+
+#VSLIDE
+
+- Transfer preparation.
+- Reporting from transfers database. <!-- .element: class="fragment" -->
+- Disk space management. <!-- .element: class="fragment" -->
+
+#HSLIDE
+
+##  Preservation: ingest
+
+Create an ingestor for Pure.
+```ruby
+ingest = Preservation::PureIngest.new
+```
+
+For each uuid, if necessary, fetch the metadata, prepare a directory in the
+ingest path and populate it with the files and JSON description file.
+
+```ruby
+ingest.prepare_dataset uuids: uuids,
+                       dir_name_scheme: :doi_short,
+                       delay: 0
+```
+
+Free up disk space for completed transfers.
+
+```ruby
+ingest.cleanup_preserved
+```
+
+#VSLIDE
+
+## Transfer-ready directory
+
+```
+.
+├── 10.17635-lancaster-researchdata-6
+│   ├── Ebola_data_Jun15.zip
+│   └── metadata
+│       └── metadata.json
+```
+
+#VSLIDE
+
+## Transfer-ready metadata
+
+```json
+[
+  {
+    "filename": "objects/Ebola_data_Jun15.zip",
+    "dc.title": "Ebolavirus evolution 2013-2015",
+    "dc.description": "Data used for analysis of selection and evolutionary rate in Zaire Ebolavirus variant Makona",
+    "dcterms.created": "2015-06-04T16:11:34.713+01:00",
+    "dcterms.available": "2015-06-04",
+    "dc.publisher": "Lancaster University",
+    "dc.identifier": "http://dx.doi.org/10.17635/lancaster/researchdata/6",
+    "dcterms.spatial": [
+      "Guinea, Sierra Leone, Liberia"
+    ],
+    "dc.creator": [
+      "Gatherer, Derek"
+    ],
+    "dc.contributor": [
+      "Robertson, David",
+      "Lovell, Simon"
+    ],
+    "dc.subject": [
+      "Ebolavirus",
+      "evolution",
+      "phylogenetics",
+      "virulence",
+      "Filoviridae",
+      "positive selection"
+    ],
+    "dcterms.license": "CC BY",
+    "related": [
+      {
+        "dc.title": "The unprecedented scale of the West African Ebola virus disease outbreak is due to environmental an$
+        "type": "Journal article",
+        "dc.identifier": "http://dx.doi.org/10.1136/ebmed-2014-110127"
+      },
+      {
+        "dc.title": "The 2014 Ebola virus disease outbreak in West Africa",
+        "type": "Journal article",
+        "dc.identifier": "http://dx.doi.org/10.1099/vir.0.067199-0"
+      }
+    ]
+  }
+]
+```
+
+#HSLIDE
+
+##  Preservation: reporting
+
+Can be used for scheduled monitoring of transfers.
+
+```ruby
+report = Preservation::IngestReport.new
+report.transfer_exception
+```
+
+#HSLIDE
+
+## Location
+
+<a href="https://rubygems.org/gems/preservation" target="_blank">RubyGems</a>
+
+<a href="https://github.com/lulibrary/preservation" target="_blank">GitHub</a>
+
+#HSLIDE
+
+## Documentation
+
+<a href="http://www.rubydoc.info/gems/preservation" target="_blank">API in YARD</a>

data/README.md

@@ -0,0 +1,88 @@
+# Preservation [![Gem Version](https://badge.fury.io/rb/preservation.svg)](https://badge.fury.io/rb/preservation) [![GitPitch](https://gitpitch.com/assets/badge.svg)](https://gitpitch.com/lulibrary/preservation/master?grs=github&t=sky)
+
+Ingest management for Archivematica's [Automation Tools](https://github.com/artefactual/automation-tools).
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'preservation'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install preservation
+
+## Usage
+
+### Configuration
+Configure Preservation. If ```log_path``` is omitted, logging (standard library) redirects to STDOUT.
+
+```ruby
+  Preservation.configure do |config|
+    config.db_path     = ENV['ARCHIVEMATICA_DB_PATH']
+    config.ingest_path = ENV['ARCHIVEMATICA_INGEST_PATH']
+    config.log_path    = ENV['PRESERVATION_LOG_PATH']
+  end
+```
+
+Configure data source.
+
+```ruby
+Puree.configure do |config|
+  config.base_url   = ENV['PURE_BASE_URL']
+  config.username   = ENV['PURE_USERNAME']
+  config.password   = ENV['PURE_PASSWORD']
+  config.basic_auth = true
+end
+```
+
+### Transfers
+
+Get some dataset UUIDs for preservation.
+
+```ruby
+c = Puree::Collection.new resource: :dataset
+minimal_metadata = c.find limit: 2,
+                          offset: 10,
+                          full: false
+uuids = []
+minimal_metadata.each do |i|
+  uuids << i['uuid']
+end
+```
+
+Create an ingestor for Pure.
+
+```ruby
+ingest = Preservation::PureIngest.new
+```
+
+For each uuid, if necessary, fetch the metadata, prepare
+a directory in the ingest path and populate it with the files and JSON description file.
+
+```ruby
+ingest.prepare_dataset uuids: uuids,
+                       dir_name_scheme: :doi_short,
+                       delay: 0
+```
+
+Free up disk space for completed transfers.
+
+```ruby
+ingest.cleanup_preserved
+```
+
+### Reporting
+Can be used for scheduled monitoring of transfers.
+
+```ruby
+report = Preservation::IngestReport.new
+report.transfer_exception
+```
+
+## Documentation
+[API in YARD](http://www.rubydoc.info/gems/preservation)

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/lib/preservation.rb

@@ -0,0 +1,24 @@
+require 'date'
+require 'fileutils'
+require 'free_disk_space'
+require 'logger'
+require 'puree'
+require 'sqlite3'
+require 'preservation/configuration'
+require 'preservation/ingest_report'
+require 'preservation/ingest'
+require 'preservation/pure_ingest'
+require 'preservation/string_util'
+require 'preservation/version'
+
+# Top level namespace
+#
+module Preservation
+
+  class << self
+
+    include Preservation::Configuration
+
+  end
+
+end

data/lib/preservation/configuration.rb

@@ -0,0 +1,15 @@
+module Preservation
+
+  # Configuration options
+  #
+  module Configuration
+
+    attr_accessor :db_path, :ingest_path, :log_path
+
+    def configure
+      yield self
+    end
+
+  end
+
+end

data/lib/preservation/ingest.rb

@@ -0,0 +1,163 @@
+module Preservation
+
+  # Base class for metadata and file management
+  #
+  class Ingest
+
+    attr_reader :logger
+
+    def initialize
+      check_ingest_path
+      setup_logger
+      setup_report
+    end
+
+    # Free up disk space for completed transfers
+    #
+    def cleanup_preserved
+      preserved = get_preserved
+      if !preserved.nil? && !preserved.empty?
+        preserved.each do |i|
+          # skip anything that has a different owner to script
+          if File.stat(i).grpowned?
+            FileUtils.remove_dir i
+            @logger.info 'Deleted ' + i
+          end
+        end
+      end
+    end
+
+
+    private
+
+    def build_wget(username, password, file_url)
+      # construct wget command with parameters
+      wget_str = ''
+      wget_str << 'wget'
+      wget_str << ' '
+      wget_str << '--user'
+      wget_str << ' '
+      wget_str << username
+      wget_str << ' '
+      wget_str << '--password'
+      wget_str << ' '
+      wget_str << '"' + password + '"'
+      wget_str << ' '
+      wget_str << file_url
+      wget_str << ' '
+      wget_str << '--no-check-certificate'
+      wget_str
+    end
+
+    def check_ingest_path
+      if Preservation.ingest_path.nil?
+        puts 'Missing ingest path'
+        exit
+      end
+    end
+
+    def setup_logger
+      if @logger.nil?
+        if Preservation.log_path.nil?
+          @logger = Logger.new STDOUT
+        else
+          # Keep data for today and the past 20 days
+          @logger = Logger.new File.new(Preservation.log_path, 'a'), 20, 'daily'
+        end
+      end
+      @logger.level = Logger::INFO
+    end
+
+    def setup_report
+      if Preservation.db_path.nil?
+        puts 'Missing db path'
+        exit
+      else
+        @report = IngestReport.new
+      end
+    end
+
+    def enough_storage_for_download?(required_bytes)
+      # scale up the required space using a multiplier
+      multiplier = 2
+      available = FreeDiskSpace.bytes('/')
+      required_bytes * multiplier < available ? true : false
+    end
+
+    def build_directory_name(metadata_record, directory_name_scheme)
+      doi = metadata_record['doi']
+      uuid = metadata_record['uuid']
+      title = metadata_record['title'].strip.gsub(' ', '-').gsub('/', '-')
+      time = Time.new
+      date = time.strftime("%Y-%m-%d")
+      time = time.strftime("%H:%M:%S")
+      join_str = '-----'
+
+      case directory_name_scheme
+        when :uuid_title
+          [uuid, title].join(join_str)
+        when :title_uuid
+          [title, uuid].join(join_str)
+        when :date_uuid_title
+          [date, uuid, title].join(join_str)
+        when :date_title_uuid
+          [date, title, uuid].join(join_str)
+        when :date_time_uuid
+          [date, time, uuid].join(join_str)
+        when :date_time_title
+          [date, time, title].join(join_str)
+        when :date_time_uuid_title
+          [date, time, uuid, title].join(join_str)
+        when :date_time_title_uuid
+          [date, time, title, uuid].join(join_str)
+        when :uuid
+          uuid
+        when :doi
+          if doi.empty?
+            return ''
+          end
+          doi.gsub('/', '-')
+        when :doi_short
+          if doi.empty?
+            return ''
+          end
+          doi_short_to_remove = 'http://dx.doi.org/'
+          short = doi.gsub(doi_short_to_remove, '')
+          short.gsub!('/', '-')
+        else
+          uuid
+      end
+    end
+
+    # time_to_preserve?
+    #
+    # @param start_utc [String]
+    # @param delay [Integer] days to wait (after modification date) before preserving
+    # @return [Boolean]
+    def time_to_preserve?(start_utc, delay)
+      now = DateTime.now
+      modified_datetime = DateTime.parse(start_utc)
+      days_since_modified = (now - modified_datetime).to_i # result in days
+      days_since_modified >= delay ? true : false
+    end
+
+    # # Collect all paths from DB where preservation has been done
+    # # @return [Array<String>]
+    def get_preserved
+      ingest_complete = @report.transfer_status(status_to_find: 'COMPLETE',
+                                                     status_presence: true)
+      preserved = []
+      ingest_complete.each do |i|
+        dir_path = Preservation.ingest_path + '/' + i['path']
+        if File.exists?(dir_path)
+          preserved << dir_path
+        end
+      end
+
+      preserved
+    end
+
+  end
+
+end
+

data/lib/preservation/ingest_report.rb

@@ -0,0 +1,172 @@
+module Preservation
+
+  # Ingest reporting
+  #
+  class IngestReport
+
+    def initialize
+      create_db_connection
+    end
+
+    # Transfers based on presence (or not) of a particular status
+    #
+    # @param status_to_find [String]
+    # @param status_presence [Boolean]
+    def transfer_status(status_to_find: nil, status_presence: true)
+      if status_presence === true
+        status_presence = '='
+      else
+        status_presence = '<>'
+      end
+
+      query = "SELECT id, uuid, hex(path) as hex_path, unit_type, status, microservice, current FROM unit WHERE status #{status_presence} ?"
+
+      # Archivematica stores path as BLOB, so need to convert path to Hex, to search for it
+      # and use hex function in DB query
+      records = []
+      @db.results_as_hash = true
+      @db.execute( query, [ status_to_find ] ) do |row|
+        id = row['id']
+        uuid = row['uuid']
+        bin_path = StringUtil.hex_to_bin row['hex_path']
+        unit_type = row['unit_type']
+        status = row['status']
+        microservice = row['microservice']
+        current = row['current']
+        o = {}
+        o['path'] = bin_path if !bin_path.empty?
+        o['unit_type'] = unit_type if !unit_type.empty?
+        o['status'] = status if !status.empty?
+        o['microservice'] = microservice if !microservice.empty?
+        o['current'] = current if current
+        o['id'] = id if id
+        o['uuid'] = uuid if !uuid.empty?
+
+        records << o
+      end
+
+      records
+    end
+
+    # Current transfer
+    #
+    # @return [Hash]
+    def transfer_current
+      query = "SELECT id, uuid, hex(path) as hex_path, unit_type, status, microservice, current FROM unit WHERE current = 1"
+
+      # Archivematica stores path as BLOB, so need to convert path to Hex, to search for it
+      # and use hex function in DB query
+      o = {}
+      @db.results_as_hash = true
+      @db.execute( query ) do |row|
+        id = row['id']
+        uuid = row['uuid']
+        bin_path = hex_to_bin row['hex_path']
+        unit_type = row['unit_type']
+        status = row['status']
+        microservice = row['microservice']
+        current = row['current']
+        o['path'] = bin_path if !bin_path.empty?
+        o['unit_type'] = unit_type if !unit_type.empty?
+        o['status'] = status if !status.empty?
+        o['microservice'] = microservice if !microservice.empty?
+        o['current'] = current if current
+        o['id'] = id if id
+        o['uuid'] = uuid if !uuid.empty?
+      end
+      o
+    end
+
+    # Count of complete transfers
+    #
+    # @return [Integer]
+    def transfer_complete_count
+      query = 'SELECT count(*) FROM unit WHERE status = ?'
+
+      status_to_find = 'COMPLETE'
+      @db.results_as_hash = true
+      @db.get_first_value( query, [status_to_find] )
+    end
+
+    # Compilation of statistics and data, with focus on exceptions
+    #
+    # @return [Hash]
+    def transfer_exception
+      incomplete = transfer_status(status_to_find: 'COMPLETE', status_presence: false)
+      failed = transfer_status(status_to_find: 'FAILED', status_presence: true)
+      current = transfer_current
+      complete_count = transfer_complete_count
+      report = {}
+      report['current'] = current if !current.empty?
+      report['failed'] = {}
+      report['failed']['count'] = failed.count
+      report['failed']['data'] = failed if !failed.empty?
+      report['incomplete'] = {}
+      report['incomplete']['count'] = incomplete.count
+      report['incomplete']['data'] = incomplete if !incomplete.empty?
+      report['complete'] = {}
+      report['complete']['count'] = complete_count if complete_count
+      report
+    end
+
+    # Is it in database?
+    # @param path_to_find [String] directory name within ingest path
+    # @return [Boolean]
+    def in_db?(path_to_find)
+      in_db = false
+
+      # Get path out of DB as a hex string
+      query = 'SELECT hex(path) FROM unit'
+
+      # Archivematica stores path as BLOB, so need to convert path to Hex, to search for it
+      # and use hex function in DB query
+      @db.execute( query ) do |row|
+        bin_path = StringUtil.hex_to_bin row[0]
+        if bin_path === path_to_find
+          in_db = true
+        end
+      end
+
+      in_db
+    end
+
+    # Has preservation been done?
+    # @param path_to_find [String] directory name within ingest path
+    # @return [Boolean]
+    def preserved?(path_to_find)
+      preserved = false
+
+      # 'ingest' value in unit_type and 'COMPLETE' value in status DB fields
+      # indicates completed
+      unit_type_to_find = 'ingest'
+      status_to_find = 'COMPLETE'
+
+      # Get path out of DB as a hex string for completed ingests
+      query = 'SELECT hex(path) FROM unit WHERE unit_type = ? AND status = ?'
+
+      # Archivematica stores path as BLOB, so need to convert path to Hex, to search for it
+      # and use hex function in DB query
+      @db.execute( query, [ unit_type_to_find, status_to_find ] ) do |row|
+        bin_path = StringUtil.hex_to_bin row[0]
+        if bin_path === path_to_find
+          preserved = true
+        end
+      end
+
+      preserved
+    end
+
+
+    private
+
+    def create_db_connection
+      if Preservation.db_path.nil?
+        puts 'Missing db_path'
+        exit
+      end
+      @db = SQLite3::Database.new Preservation.db_path
+    end
+
+  end
+
+end

data/lib/preservation/pure_ingest.rb

@@ -0,0 +1,188 @@
+module Preservation
+
+  # Ingest for Pure
+  #
+  class PureIngest < Ingest
+
+    def initialize
+      super
+    end
+
+    # For each uuid, if necessary, fetch the metadata,
+    # prepare a directory in the ingest path and populate it with the files and
+    # JSON description file.
+    #
+    # @param uuids [Array<String>] uuids to preserve
+    # @param dir_name_scheme [Symbol] method to make directory name
+    # @param delay [Integer] days to wait (after modification date) before preserving
+    def prepare_dataset(uuids: [],
+                        dir_name_scheme: :uuid,
+                        delay: 0)
+      dir_base_path = Preservation.ingest_path
+
+      uuids.each do |uuid|
+        dataset = Puree::Dataset.new
+        dataset.find uuid: uuid
+        d = dataset.metadata
+        if d.empty?
+          @logger.info 'No metadata for ' + uuid
+          next
+        end
+        # configurable to become more human-readable
+        dir_name = build_directory_name(d, dir_name_scheme)
+
+        # continue only if dir_name is not empty (e.g. because there was no DOI)
+        # continue only if there is no DB entry
+        # continue only if the dataset has a DOI
+        # continue only if there are files for this resource
+        # continue only if it is time to preserve
+        if !dir_name.nil? &&
+           !dir_name.empty? &&
+           !@report.in_db?(dir_name) &&
+           !d['doi'].empty? &&
+           !d['file'].empty? &&
+           time_to_preserve?(d['modified'], delay)
+
+          dir_file_path = dir_base_path + '/' + dir_name
+          dir_metadata_path = dir_file_path + '/metadata/'
+          metadata_filename = dir_metadata_path + 'metadata.json'
+
+          # calculate total size of data files
+          download_storage_required = 0
+          d['file'].each { |i| download_storage_required += i['size'].to_i }
+
+          # do we have enough space in filesystem to fetch data files?
+          if enough_storage_for_download? download_storage_required
+            # @logger.info 'Sufficient disk space for ' + dir_file_path
+          else
+            @logger.error 'Insufficient disk space to store files fetched from Pure. Skipping ' + dir_file_path
+            next
+          end
+
+          # has metadata file been created? if so, files and metadata are in place
+          # continue only if files not present in ingest location
+          if !File.size? metadata_filename
+
+            @logger.info 'Preparing ' + dir_name + ', Pure UUID ' + d['uuid']
+
+            data = []
+            d['file'].each do |f|
+              o = package_dataset_metadata d, f
+              data << o
+              wget_str = build_wget Puree.username,
+                                    Puree.password,
+                                    f['url']
+
+              Dir.mkdir(dir_file_path) if !Dir.exists?(dir_file_path)
+
+              # fetch the file
+              Dir.chdir(dir_file_path) do
+                # puts 'Changing dir to ' + Dir.pwd
+                # puts 'Size of ' + f['name'] + ' is ' + File.size(f['name']).to_s
+                if File.size?(f['name'])
+                  # puts 'Should be deleting ' + f['name']
+                  File.delete(f['name'])
+                end
+                # puts f['name'] + ' missing or empty'
+                # puts wget_str
+                `#{wget_str}`
+              end
+            end
+
+            Dir.mkdir(dir_metadata_path) if !Dir.exists?(dir_metadata_path)
+
+            pretty = JSON.pretty_generate( data, :indent => '  ')
+            # puts pretty
+            File.write(metadata_filename,pretty)
+            @logger.info 'Created ' + metadata_filename
+          end
+        else
+          @logger.info 'Skipping ' + dir_name + ', Pure UUID ' + d['uuid']
+        end
+      end
+    end
+
+    private
+
+    def package_dataset_metadata(d, f)
+        o = {}
+        o['filename'] = 'objects/' + f['name']
+        o['dc.title'] = d['title']
+        if !d['description'].empty?
+          o['dc.description'] = d['description']
+        end
+        o['dcterms.created'] = d['created']
+        if !d['available']['year'].empty?
+          o['dcterms.available'] = Puree::Date.iso(d['available'])
+        end
+        o['dc.publisher'] = d['publisher']
+        if !d['doi'].empty?
+          o['dc.identifier'] = d['doi']
+        end
+        if !d['spatial'].empty?
+          o['dcterms.spatial'] = d['spatial']
+        end
+        if !d['temporal']['start']['year'].empty?
+          temporal_range = ''
+          temporal_range << Puree::Date.iso(d['temporal']['start'])
+          if !d['temporal']['end']['year'].empty?
+            temporal_range << '/'
+            temporal_range << Puree::Date.iso(d['temporal']['end'])
+          end
+          o['dcterms.temporal'] = temporal_range
+        end
+        creators = []
+        contributors = []
+        person_types = %w(internal external other)
+        person_types.each do |person_type|
+          d['person'][person_type].each do |i|
+            if i['role'] == 'Creator'
+              creator = i['name']['last'] + ', ' + i['name']['first']
+              creators << creator
+            end
+            if i['role'] == 'Contributor'
+              contributor = i['name']['last'] + ', ' + i['name']['first']
+              contributors << contributor
+            end
+          end
+        end
+        o['dc.creator'] = creators
+        if !contributors.empty?
+          o['dc.contributor'] = contributors
+        end
+        keywords = []
+        d['keyword'].each { |i|
+          keywords << i
+        }
+        if !keywords.empty?
+          o['dc.subject'] = keywords
+        end
+        if !f['license']['name'].empty?
+          o['dcterms.license'] = f['license']['name']
+        end
+        # o['dc.format'] = f['mime']
+
+        related = []
+        publications = d['publication']
+        publications.each do |i|
+          o_related = {}
+          o_related['dc.title'] = i['title']
+          o_related['type'] = i['type']
+          pub = Puree::Publication.new
+          pub.find uuid: i['uuid']
+          doi = pub.doi
+          if doi
+            o_related['dc.identifier'] = doi
+          end
+          related << o_related
+        end
+        if !related.empty?
+          o['related'] = related
+        end
+
+        o
+    end
+
+  end
+
+end

data/lib/preservation/string_util.rb

@@ -0,0 +1,19 @@
+module Preservation
+
+  # String utilities
+  #
+  module StringUtil
+    # Binary to hexadecimal
+    #
+    def self.bin_to_hex(s)
+      s.each_byte.map { |b| b.to_s(16) }.join
+    end
+
+    # Hexadecimal to binary
+    def self.hex_to_bin(s)
+      s.scan(/../).map { |x| x.hex.chr }.join
+    end
+
+  end
+
+end

data/lib/preservation/version.rb

@@ -0,0 +1,5 @@
+module Preservation
+  # Semantic version number
+  #
+  VERSION = "0.1.0"
+end

data/preservation.gemspec

@@ -0,0 +1,26 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'preservation/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "preservation"
+  spec.version       = Preservation::VERSION
+  spec.authors       = ["Adrian Albin-Clark"]
+  spec.email         = ["a.albin-clark@lancaster.ac.uk"]
+  spec.summary       = %q{Ingest management for Archivematica's Automation Tools.}
+  spec.description   = %q{Transfer preparation, reporting and disk space management for Archivematica's Automation Tools.}
+  spec.homepage      = "https://rubygems.org/gems/preservation"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.required_ruby_version = '~> 2.1'
+
+  spec.add_runtime_dependency 'free_disk_space', '~> 1.0'
+  spec.add_runtime_dependency 'puree', '~> 0.17'
+  spec.add_runtime_dependency'sqlite3', '~> 1.3'
+end

metadata

@@ -0,0 +1,103 @@
+--- !ruby/object:Gem::Specification
+name: preservation
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Adrian Albin-Clark
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-09-13 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: free_disk_space
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: puree
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.17'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.17'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+description: Transfer preparation, reporting and disk space management for Archivematica's
+  Automation Tools.
+email:
+- a.albin-clark@lancaster.ac.uk
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- CHANGELOG.md
+- Gemfile
+- LICENSE.txt
+- PITCHME.md
+- README.md
+- Rakefile
+- lib/preservation.rb
+- lib/preservation/configuration.rb
+- lib/preservation/ingest.rb
+- lib/preservation/ingest_report.rb
+- lib/preservation/pure_ingest.rb
+- lib/preservation/string_util.rb
+- lib/preservation/version.rb
+- preservation.gemspec
+homepage: https://rubygems.org/gems/preservation
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - "~>"
+    - !ruby/object:Gem::Version
+      version: '2.1'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.2.2
+signing_key: 
+specification_version: 4
+summary: Ingest management for Archivematica's Automation Tools.
+test_files: []
+has_rdoc: