preservation 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dfcf307b70473079a60f2801c6bd11e4f4c289d8
-  data.tar.gz: 0e13efac8904ccd96fc690644520e20f388a4fce
+  metadata.gz: 37837b8796fc9b31c0c135b966de5837fd99aba9
+  data.tar.gz: eb8bd1f506a30b8035b18d541c9fabc84c67fe29
 SHA512:
-  metadata.gz: 939f44e9a24177232e900953f4b59b64cfe62e0bce056765444dd9a726b64b441cae7b8fc6b3cea233ef9febd732a9a27f0759992cafe4e8494213edc947fffa
-  data.tar.gz: 2a86f065c50cf43c8ea7bcaa707d278f05b7b73b2be53cbebad47e50f38ffa3e8324dc88a71ded669495e6c7f719ca7a7dbd8943da3bcf43a186f6f56df40daf
+  metadata.gz: 5d1f1e0ea0408952329524d1653da892eeb61f1ab1a3ef4847ee4520b0255c464f7a18c3969f5f3c96b5e0ae69d1b05ae22ea63eb91ced76c75288d7527cd52c
+  data.tar.gz: b559abab2a467dacb0cb8afcd531b9cea7c8415ce3b08c7b653d4711631bf4c30ef40d992e806150b83c17664d3fbe7ec734cc80de2011e3032021643924d06f

data/CHANGELOG.md

@@ -4,6 +4,11 @@ This project adheres to [Semantic Versioning](http://semver.org/).
 
 ## Unreleased
 
+## 0.2.0 - 2016-09-17
+### Changed
+- Singular uuid rather than an array of uuids as parameter for transfer preparation.
+- Modules, classes and API.
+
 ## 0.1.0 - 2016-09-13
 ### Added
 - Transfer preparation.

data/PITCHME.md

@@ -1,10 +1,9 @@
 #HSLIDE
 
 ## Rationale
-Archivematica's [Automation Tools](https://github.com/artefactual/automation-tools)
+Archivematica's <a href="https://github.com/artefactual/automation-tools" target="_blank">Automation Tools</a>
 work with files and descriptive metadata which must be provided in a certain way.
 
-
 #HSLIDE
 
 ## Preservation: a way to manage ingest
@@ -17,32 +16,33 @@ work with files and descriptive metadata which must be provided in a certain way
 
 #HSLIDE
 
-##  Preservation: ingest
+##  Preservation: transfer
+
+Create a transfer using the Pure Research Information System as a data source.
 
-Create an ingestor for Pure.
 ```ruby
-ingest = Preservation::PureIngest.new
+transfer = Preservation::Transfer::Pure.new base_url:   ENV['PURE_BASE_URL'],
+                                            username:   ENV['PURE_USERNAME'],
+                                            password:   ENV['PURE_PASSWORD'],
+                                            basic_auth: true
 ```
 
-For each uuid, if necessary, fetch the metadata, prepare a directory in the
-ingest path and populate it with the files and JSON description file.
+For a Pure dataset, if necessary, fetch the metadata, prepare
+a directory in the ingest path and populate it with the files and JSON description file.
 
 ```ruby
-ingest.prepare_dataset uuids: uuids,
-                       dir_name_scheme: :doi_short,
-                       delay: 0
+transfer.prepare_dataset uuid: 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'
 ```
 
-Free up disk space for completed transfers.
+Free up disk space for completed transfers. Can be done at any time.
 
 ```ruby
-ingest.cleanup_preserved
+Preservation::Storage.cleanup
 ```
 
 #VSLIDE
 
 ## Transfer-ready directory
-
 ```
 .
 ├── 10.17635-lancaster-researchdata-6
@@ -86,7 +86,7 @@ ingest.cleanup_preserved
     "dcterms.license": "CC BY",
     "related": [
       {
-        "dc.title": "The unprecedented scale of the West African Ebola virus disease outbreak is due to environmental an$
+        "dc.title": "The unprecedented scale of the West African Ebola virus disease outbreak is due to environmental and sociological factors, not special attributes of the currently circulating strain of the virus",
         "type": "Journal article",
         "dc.identifier": "http://dx.doi.org/10.1136/ebmed-2014-110127"
       },
@@ -107,8 +107,7 @@ ingest.cleanup_preserved
 Can be used for scheduled monitoring of transfers.
 
 ```ruby
-report = Preservation::IngestReport.new
-report.transfer_exception
+Preservation::Report::Transfer.exception
 ```
 
 #HSLIDE
@@ -123,4 +122,6 @@ report.transfer_exception
 
 ## Documentation
 
-<a href="http://www.rubydoc.info/gems/preservation" target="_blank">API in YARD</a>
+<a href="http://www.rubydoc.info/gems/preservation" target="_blank">API in YARD</a>
+
+<a href="https://aalbinclark.gitbooks.io/preservation" target="_blank">Detailed usage in GitBook</a>

data/README.md

@@ -1,6 +1,6 @@
 # Preservation [![Gem Version](https://badge.fury.io/rb/preservation.svg)](https://badge.fury.io/rb/preservation) [![GitPitch](https://gitpitch.com/assets/badge.svg)](https://gitpitch.com/lulibrary/preservation/master?grs=github&t=sky)
 
-Ingest management for Archivematica's [Automation Tools](https://github.com/artefactual/automation-tools).
+Extraction and Transformation for Loading by Archivematica's Automation Tools.
 
 ## Installation
 
@@ -19,70 +19,48 @@ Or install it yourself as:
 ## Usage
 
 ### Configuration
-Configure Preservation. If ```log_path``` is omitted, logging (standard library) redirects to STDOUT.
+Configure Preservation. If ```log_path``` is omitted, logging (standard library) writes to STDOUT.
 
 ```ruby
-  Preservation.configure do |config|
-    config.db_path     = ENV['ARCHIVEMATICA_DB_PATH']
-    config.ingest_path = ENV['ARCHIVEMATICA_INGEST_PATH']
-    config.log_path    = ENV['PRESERVATION_LOG_PATH']
-  end
-```
-
-Configure data source.
-
-```ruby
-Puree.configure do |config|
-  config.base_url   = ENV['PURE_BASE_URL']
-  config.username   = ENV['PURE_USERNAME']
-  config.password   = ENV['PURE_PASSWORD']
-  config.basic_auth = true
+Preservation.configure do |config|
+  config.db_path     = ENV['ARCHIVEMATICA_DB_PATH']
+  config.ingest_path = ENV['ARCHIVEMATICA_INGEST_PATH']
+  config.log_path    = ENV['PRESERVATION_LOG_PATH']
 end
 ```
 
-### Transfers
-
-Get some dataset UUIDs for preservation.
 
-```ruby
-c = Puree::Collection.new resource: :dataset
-minimal_metadata = c.find limit: 2,
-                          offset: 10,
-                          full: false
-uuids = []
-minimal_metadata.each do |i|
-  uuids << i['uuid']
-end
-```
-
-Create an ingestor for Pure.
+### Transfer
+Create a transfer using the Pure Research Information System as a data source.
 
 ```ruby
-ingest = Preservation::PureIngest.new
+transfer = Preservation::Transfer::Pure.new base_url:   ENV['PURE_BASE_URL'],
+                                            username:   ENV['PURE_USERNAME'],
+                                            password:   ENV['PURE_PASSWORD'],
+                                            basic_auth: true
 ```
 
-For each uuid, if necessary, fetch the metadata, prepare
+For a Pure dataset, if necessary, fetch the metadata, prepare
 a directory in the ingest path and populate it with the files and JSON description file.
 
 ```ruby
-ingest.prepare_dataset uuids: uuids,
-                       dir_name_scheme: :doi_short,
-                       delay: 0
+transfer.prepare_dataset uuid: 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'
 ```
 
-Free up disk space for completed transfers.
+Free up disk space for completed transfers. Can be done at any time.
 
 ```ruby
-ingest.cleanup_preserved
+Preservation::Storage.cleanup
 ```
 
-### Reporting
+### Report
 Can be used for scheduled monitoring of transfers.
 
 ```ruby
-report = Preservation::IngestReport.new
-report.transfer_exception
+Preservation::Report::Transfer.exception
 ```
 
 ## Documentation
-[API in YARD](http://www.rubydoc.info/gems/preservation)
+[API in YARD](http://www.rubydoc.info/gems/preservation)
+
+[Detailed usage in GitBook](https://aalbinclark.gitbooks.io/preservation)

data/lib/preservation.rb

@@ -5,10 +5,14 @@ require 'logger'
 require 'puree'
 require 'sqlite3'
 require 'preservation/configuration'
-require 'preservation/ingest_report'
+require 'preservation/report/database'
+require 'preservation/report/transfer'
+require 'preservation/conversion'
 require 'preservation/ingest'
-require 'preservation/pure_ingest'
-require 'preservation/string_util'
+require 'preservation/builder'
+require 'preservation/storage'
+require 'preservation/temporal'
+require 'preservation/transfer/pure'
 require 'preservation/version'
 
 # Top level namespace

data/lib/preservation/builder.rb

@@ -0,0 +1,84 @@
+module Preservation
+
+  # Builder
+  #
+  module Builder
+
+    # Build wget string
+    #
+    # @param username [String]
+    # @param password [String]
+    # @param file_url [String]
+    # @return [String]
+    def self.build_wget(username, password, file_url)
+      # construct wget command with parameters
+      wget_str = ''
+      wget_str << 'wget'
+      wget_str << ' '
+      wget_str << '--user'
+      wget_str << ' '
+      wget_str << username
+      wget_str << ' '
+      wget_str << '--password'
+      wget_str << ' '
+      wget_str << '"' + password + '"'
+      wget_str << ' '
+      wget_str << file_url
+      wget_str << ' '
+      wget_str << '--no-check-certificate'
+      wget_str
+    end
+
+    # Build directory name
+    #
+    # @param metadata record [Hash]
+    # @param directory_name_scheme [Symbol]
+    # @return [String]
+    def self.build_directory_name(metadata_record, directory_name_scheme)
+      doi = metadata_record['doi']
+      uuid = metadata_record['uuid']
+      title = metadata_record['title'].strip.gsub(' ', '-').gsub('/', '-')
+      time = Time.new
+      date = time.strftime("%Y-%m-%d")
+      time = time.strftime("%H:%M:%S")
+      join_str = '-----'
+
+      case directory_name_scheme
+        when :uuid_title
+          [uuid, title].join(join_str)
+        when :title_uuid
+          [title, uuid].join(join_str)
+        when :date_uuid_title
+          [date, uuid, title].join(join_str)
+        when :date_title_uuid
+          [date, title, uuid].join(join_str)
+        when :date_time_uuid
+          [date, time, uuid].join(join_str)
+        when :date_time_title
+          [date, time, title].join(join_str)
+        when :date_time_uuid_title
+          [date, time, uuid, title].join(join_str)
+        when :date_time_title_uuid
+          [date, time, title, uuid].join(join_str)
+        when :uuid
+          uuid
+        when :doi
+          if doi.empty?
+            return ''
+          end
+          doi.gsub('/', '-')
+        when :doi_short
+          if doi.empty?
+            return ''
+          end
+          doi_short_to_remove = 'http://dx.doi.org/'
+          short = doi.gsub(doi_short_to_remove, '')
+          short.gsub!('/', '-')
+        else
+          uuid
+      end
+    end
+
+  end
+
+end

data/lib/preservation/{string_util.rb → conversion.rb}

@@ -1,15 +1,20 @@
 module Preservation
 
-  # String utilities
+  # Conversion
   #
-  module StringUtil
+  module Conversion
     # Binary to hexadecimal
     #
+    # @param [Binary String]
+    # @return [Hexadecimal String]
     def self.bin_to_hex(s)
       s.each_byte.map { |b| b.to_s(16) }.join
     end
 
     # Hexadecimal to binary
+    #
+    # @param [Hexadecimal String]
+    # @return [Binary String]
     def self.hex_to_bin(s)
       s.scan(/../).map { |x| x.hex.chr }.join
     end

data/lib/preservation/ingest.rb

@@ -1,57 +1,21 @@
 module Preservation
 
-  # Base class for metadata and file management
+  # Ingest
   #
   class Ingest
 
     attr_reader :logger
 
     def initialize
-      check_ingest_path
       setup_logger
-      setup_report
-    end
-
-    # Free up disk space for completed transfers
-    #
-    def cleanup_preserved
-      preserved = get_preserved
-      if !preserved.nil? && !preserved.empty?
-        preserved.each do |i|
-          # skip anything that has a different owner to script
-          if File.stat(i).grpowned?
-            FileUtils.remove_dir i
-            @logger.info 'Deleted ' + i
-          end
-        end
-      end
-    end
-
+      check_ingest_path
+     end
 
     private
 
-    def build_wget(username, password, file_url)
-      # construct wget command with parameters
-      wget_str = ''
-      wget_str << 'wget'
-      wget_str << ' '
-      wget_str << '--user'
-      wget_str << ' '
-      wget_str << username
-      wget_str << ' '
-      wget_str << '--password'
-      wget_str << ' '
-      wget_str << '"' + password + '"'
-      wget_str << ' '
-      wget_str << file_url
-      wget_str << ' '
-      wget_str << '--no-check-certificate'
-      wget_str
-    end
-
     def check_ingest_path
       if Preservation.ingest_path.nil?
-        puts 'Missing ingest path'
+        @logger.error 'Missing ingest path'
         exit
       end
     end
@@ -68,95 +32,6 @@ module Preservation
       @logger.level = Logger::INFO
     end
 
-    def setup_report
-      if Preservation.db_path.nil?
-        puts 'Missing db path'
-        exit
-      else
-        @report = IngestReport.new
-      end
-    end
-
-    def enough_storage_for_download?(required_bytes)
-      # scale up the required space using a multiplier
-      multiplier = 2
-      available = FreeDiskSpace.bytes('/')
-      required_bytes * multiplier < available ? true : false
-    end
-
-    def build_directory_name(metadata_record, directory_name_scheme)
-      doi = metadata_record['doi']
-      uuid = metadata_record['uuid']
-      title = metadata_record['title'].strip.gsub(' ', '-').gsub('/', '-')
-      time = Time.new
-      date = time.strftime("%Y-%m-%d")
-      time = time.strftime("%H:%M:%S")
-      join_str = '-----'
-
-      case directory_name_scheme
-        when :uuid_title
-          [uuid, title].join(join_str)
-        when :title_uuid
-          [title, uuid].join(join_str)
-        when :date_uuid_title
-          [date, uuid, title].join(join_str)
-        when :date_title_uuid
-          [date, title, uuid].join(join_str)
-        when :date_time_uuid
-          [date, time, uuid].join(join_str)
-        when :date_time_title
-          [date, time, title].join(join_str)
-        when :date_time_uuid_title
-          [date, time, uuid, title].join(join_str)
-        when :date_time_title_uuid
-          [date, time, title, uuid].join(join_str)
-        when :uuid
-          uuid
-        when :doi
-          if doi.empty?
-            return ''
-          end
-          doi.gsub('/', '-')
-        when :doi_short
-          if doi.empty?
-            return ''
-          end
-          doi_short_to_remove = 'http://dx.doi.org/'
-          short = doi.gsub(doi_short_to_remove, '')
-          short.gsub!('/', '-')
-        else
-          uuid
-      end
-    end
-
-    # time_to_preserve?
-    #
-    # @param start_utc [String]
-    # @param delay [Integer] days to wait (after modification date) before preserving
-    # @return [Boolean]
-    def time_to_preserve?(start_utc, delay)
-      now = DateTime.now
-      modified_datetime = DateTime.parse(start_utc)
-      days_since_modified = (now - modified_datetime).to_i # result in days
-      days_since_modified >= delay ? true : false
-    end
-
-    # # Collect all paths from DB where preservation has been done
-    # # @return [Array<String>]
-    def get_preserved
-      ingest_complete = @report.transfer_status(status_to_find: 'COMPLETE',
-                                                     status_presence: true)
-      preserved = []
-      ingest_complete.each do |i|
-        dir_path = Preservation.ingest_path + '/' + i['path']
-        if File.exists?(dir_path)
-          preserved << dir_path
-        end
-      end
-
-      preserved
-    end
-
   end
 
 end