fedora-migrate 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1cd495240ef6ef07dafbfa15720b1c5eca3a4146
-  data.tar.gz: 4d59a16932745f6082133cf3a5492a024e499f84
+  metadata.gz: a1c6f75dc48d80e784199985d948420c2c449de4
+  data.tar.gz: 0002dcb8c3e0508510359bc427c70cd2cf0b1f76
 SHA512:
-  metadata.gz: 3a906945035f3966d5c5ecb24668081da701e52a5cec8f881b0b75d3e938ee72a53354008315911e1dd3914793311f9efb79991aeb78199ef210f659e6b9c111
-  data.tar.gz: 18a47d1a58619bd9a57f4489537f79a35655f184eac5f3a47e2424f5feef646946e407d1bdc6c52504cd529d42f7f61bfa98935dab8f8e98c83e881178660ff6
+  metadata.gz: cc62419d71580ecda28950eea55ada46b397d517730eff12f3d3bdbfb3b184565a1499f8ec2e219d24f61395b7be42ea2f357b18ec772d4887b6706786a97301
+  data.tar.gz: e2adbf133dc201a4176958a53b83fe9dfc32351dfd59a72651e73ac527b4be01adf8ee762e76ac6233bb118b88daca67af1091ac4e09291fb2b888d28a56d4f6

data/.gitignore

@@ -14,3 +14,4 @@
 mkmf.log
 jetty
 .ruby-version
+.rvmrc

data/Gemfile

@@ -5,3 +5,4 @@ gemspec
 
 gem 'byebug'
 gem 'pry'
+gem 'http_logger'

data/LICENSE

@@ -0,0 +1,14 @@
+Copyright © 2015 The Pennsylvania State University
+Additional copyright may be held by others, as reflected in the commit history.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -1,22 +1,211 @@
-# FedoraMigrate
+# FedoraMigrate [![Version](https://badge.fury.io/gh/projecthydra-labs%2Ffedora-migrate.png)](http://badge.fury.io/gh/projecthydra-labs%2Ffedora-migrate) [![Build Status](https://travis-ci.org/projecthydra-labs/fedora-migrate.png?branch=master)](https://travis-ci.org/projecthydra-labs/fedora-migrate)
 
 Migrates content from a Fedora3 repository to a Fedora4 one.
 
 ## Status
 
-Very alpha. This has been tested against Penn State's existing Scholarsohere applications, as well
-as generic Sufia applications.
+This has been tested against Penn State's existing Scholarsphere applications, as well as generic Sufia applications.
+Other uses are presently unknown.
 
-## TODOs
+## Overview
 
-See the list of issues.
+FedoraMigrate iterates over your existing Fedora3 application using the Rubydora gem. For each object it finds, it
+creates a new object with the same id in Fedora4 and proceeds to migrate each datastream, including versions if
+they are defined, and verifies the checksum of each. Permissions and relationships are migrated as well but using
+different procedures due to the changes in Fedora4.
+
+The entire migration process takes place in two steps. In the first, all objects, including datastreams and permissions,
+are copied over to Fedora4; in the second, relationships are migrated.
+
+## Requirements
+
+1. A working Hydra application using Fedora4
+2. An existing Fedora3 instance
+3. All models defined in your Hydra/Fedora4 application
+
+## Usage
+
+Add the fedora-migrate gem to your existing Fedora4-based Hydra head
+
+    gem 'fedora-migrate'
+
+Then run `bundle update`
+
+Create a `config/fedora3.yml` file and point it to your current Fedora3 repository
+
+    development:
+      user: fedoraAdmin
+      password: fedoraAdmin
+      url: http://localhost:8983/fedora3
+    test:
+      user: fedoraAdmin
+      password: fedoraAdmin
+      url: http://localhost:8983/fedora3
+    production:
+      user: fedoraAdmin
+      password: fedoraAdmin
+      url: http://localhost:8983/fedora3
+
+Create a rake task to migrate your repository. You can use the following, taken from `lib/tasks/fedora-migrate.rake`,
+as an example:
+
+``` ruby
+desc "Migrate all my objects"
+task migrate: :environment do
+  results = FedoraMigrate.migrate_repository(namespace: "mynamespace")
+  puts results
+end
+```
+
+Run the task
+
+    $ bundle exec rake migrate
+
+By default, messages are logged to your Rails environment logs.
+
+## Configuration
+
+FedoraMigrate uses your existing Hydra/Fedora4 application as the basis for migrating objects. For example,
+given the model
+
+``` ruby
+class MyModel < ActiveFedora::Base
+  contains "content", class_name: "ActiveFedora::File"
+  contains "thumbnail", class_name: "ActiveFedora::File"
+end
+```
+
+When the migrator finds an object in your Fedora3 repository that has the name _MyModel_ it attempts to instantiate the
+object `MyModel` in the context of your Hydra application. Only the datastreams, or files, that are defined in the model will
+be migrated from Fedora3. This means if your Fedora3 object has the datastream "special" but it is not in your Hydra
+model, it will not be migrated. DC datastreams are not migrated by default, and RELS-EXT and rightsMetdata datastreams are treated
+differently. See [FedoraMigrate::RelsExtDatastreamMover](lib/fedora_migrate/rels_ext_datastream_mover.rb) and
+[FedoraMigrate::PermissionsMover](lib/fedora_migrate/permissions_mover.rb).
+
+If your model contains a file or datastream that is versioned, then all versions of that datastream will be migrated from
+Fedora3. If the model does not define something as versioned, yet the Fedora3 datastream is versioned, then only the current
+version will be migrated to Fedora4.
+
+### RDF Conversion
+
+If you elect to do so, FedoraMigrate will attempt to convert ActiveFedora::NtriplesRDFDatastream objects into RDF properties
+defined on your object. You can configure this as an option passed to the migrator.
+
+``` ruby
+FedoraMigrate.migrate_repository(namespace: "mynamespace", options: {convert: "descMetadata"})
+```
+
+However, you are required to define any and all RDF properties on your object in Hydra. For example, given
+
+``` ruby
+class RDFObject < ActiveFedora::Base
+  property :title, predicate: ::RDF::DC.title do |index|
+    index.as :stored_searchable, :facetable
+  end
+  contains "content", class_name: "ActiveFedora::File"
+  contains "thumbnail", class_name: "ActiveFedora::File"
+end
+```
+
+If your descMetadata RDF datastream in Fedora3 contains the triple
+
+    <info:fedora/mynamespace:xp68km39w> <http://purl.org/dc/terms/title> "My Title" .
+
+Then FedoraMigrate will define that property on your Fedora4 object using the DC term.
+
+### Object Migration
+
+By default, FedoraMigrate will use [FedoraMigrate::TargetConstructor](lib/fedora_migrate/target_constructor.rb)
+to find a model in your Hydra application that matches
+the Fedora3 source object. The constructor is designed to work with Hydra applications. If need be, you can override this
+class by creating a new one that determines a model name based on your own criteria.
+
+``` ruby
+module FedoraMigrate
+  class TargetConstructor
+
+    attr_accessor :candidates, :target
+
+    def initialize candidates
+      @candidates = candidates
+    end
+
+    def build
+      # set target to whichever model you need based on candidates
+      return self
+    end
+  end
+end
+```
+
+You can also opt to provide your own model, if you wish, by passing it as a second argument to the object mover class.
+
+``` ruby
+source = FedoraMigrate.source.connection.find("mynamespace:rb68xc089")
+mover = FedoraMigrate::ObjectMover.new source, CustomObject.new
+mover.migrate
+```
+
+### Configuration Hooks
+
+Because the migration process will be different for each user, overridable methods are placed before and after each step in the
+migration process. These can be used if your source or target objects need additional preparation before they can be
+migrated. A good example is in Sufia, where a depositor must be applied before the object can be saved.
+
+To use the hooks, simply define them in your migration task
+
+``` ruby
+module FedoraMigrate::Hooks
+
+  # Both @source and @target are available, as the Rubydora object and ActiveFedora model, respectively
+
+  # Apply depositor metadata before you migrate an object
+  def before_object_migration
+    xml = Nokogiri::XML(source.datastreams["properties"].content)
+    target.apply_depositor_metadata xml.xpath("//depositor").text
+  end
+
+  def after_object_migration
+    # additional actions as needed
+  end
+
+end
+
+desc "Migrate all my objects"
+task migrate: :environment do
+  results = FedoraMigrate.migrate_repository(namespace: "mynamespace", options: {convert: "descMetadata"})
+  puts results
+end
+```
+
+## Testing
+
+Execute `bundle exec rake` to run the test suite.
+
+    $ bundle exec rake jetty:clean jetty:start
+    $ bundle exec rake fixtures:load
+    $ bundle exec rspec
+
+This will run all the spec tests and leave jetty running if you wish to run specific tests.
+
+If you have sample objects that you feel should be used as relevant testing examples, please add them to
+`spec/fixtures/objects` and re-run the tests. Sample objects should be exported from existing Fedora3
+repositories as foxml files using the "archive" option. This can be done via the admin web interface,
+[http://localhost:8983/fedora3/admin](http://localhost:8983/fedora3/admin), or using 
+`FEDORA_HOME/client/bin/fedora-export.sh`.
+
+*Note that the script option may only work under full installs of Fedora3 and not hydra-jetty.*
+
+## TODOs and Reporting Errors
+
+See the list of issues for current bugs and feature needs. Add your own as needed.
 
 ## Contributing
 
 ### Hydra Developers
 
 For Hydra developers, or anyone with a signed CLA, please clone the repo and submit PRs via
-topic branches. If you don't have rights to projecthydra-labs and do have a signed
+feature branches. If you don't have rights to projecthydra-labs and do have a signed
 CLA, please send a note to hydra-tech@googlegroups.com.
 
 1. Clone it
@@ -29,3 +218,9 @@ CLA, please send a note to hydra-tech@googlegroups.com.
 
 Anyone is welcome to use this software and report issues.
 In order to merge any work contributed, you'll need to sign a contributor license agreement.
+For more information on signing a CLA, please contact `legal@projecthyra.org`
+# Project Hydra
+This software has been developed by and is brought to you by the Hydra community.  Learn more at the
+[Project Hydra website](http://projecthydra.org)
+
+![Project Hydra Logo](https://github.com/uvalib/libra-oa/blob/a6564a9e5c13b7873dc883367f5e307bf715d6cf/public/images/powered_by_hydra.png?raw=true)

data/config/jetty.yml

@@ -2,5 +2,5 @@ default:
   startup_wait: 90
   jetty_port: 8983
   java_opts:
-    - "-Xmx256m" 
-    - "-XX:MaxPermSize=128m"
+    - "-Xmx512m" 
+    - "-XX:MaxPermSize=256m"

data/fedora-migrate.gemspec

@@ -18,13 +18,14 @@ Gem::Specification.new do |spec|
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
 
-  spec.add_dependency "rubydora", "~> 1.8" 
-  spec.add_dependency "hydra-head", "~> 9.0.0.beta1"
-
+  spec.add_dependency "hydra-head", "~> 9.0"
+  spec.add_dependency "rubydora", "~> 1.8"
+  spec.add_dependency "rchardet"
+  
   spec.add_development_dependency "bundler", "~> 1.7"
+  spec.add_development_dependency "equivalent-xml"
+  spec.add_development_dependency "hydra-collections", "~> 4.0"
+  spec.add_development_dependency "jettywrapper"
   spec.add_development_dependency "rake", "~> 10.0"
   spec.add_development_dependency "rspec"
-  spec.add_development_dependency "jettywrapper"
-  spec.add_development_dependency "equivalent-xml"
-  spec.add_development_dependency "sufia", "~> 6.0.0.beta1"
 end

data/lib/fedora-migrate.rb

@@ -1,36 +1,19 @@
 require "fedora_migrate/version"
 require "active_support"
 require "active_fedora"
-require "hydra-core"
+require "hydra/head"
+require "rubydora"
 
 # Loads rake tasks
 Dir[File.expand_path(File.join(File.dirname(__FILE__),"tasks/*.rake"))].each { |ext| load ext } if defined?(Rake)
 
-# Shenanigans because we're not in a Rails environment and we need
-# Hydra::AccessControls
-Hydra::Engine.config.autoload_paths.each { |path| $LOAD_PATH.unshift path }
-# in gem version 2.4, .find_by_name isn't pulling up gems given in the Gemfile
-# as opposed to those in the gemspec file.
-# This is a workaround:
-Gem::Specification.all.each do |g|
-  HAC_DIR = g.gem_dir if g.name.match("hydra-access-controls")
-end
-require HAC_DIR+'/app/vocabularies/acl'
-require HAC_DIR+'/app/vocabularies/hydra/acl'
-require HAC_DIR+'/app/models/role_mapper'
-require HAC_DIR+'/app/models/ability'
-require HAC_DIR+'/app/models/hydra/access_controls/access_control_list'
-require HAC_DIR+'/app/models/hydra/access_controls/permission'
-require HAC_DIR+'/app/models/hydra/access_controls/embargo'
-require HAC_DIR+'/app/models/hydra/access_controls/lease'
-require HAC_DIR+'/app/services/hydra/lease_service'
-require HAC_DIR+'/app/services/hydra/embargo_service'
-require HAC_DIR+'/app/validators/hydra/future_date_validator'
-
 module FedoraMigrate
   extend ActiveSupport::Autoload
 
+  autoload :ContentMover
   autoload :DatastreamMover
+  autoload :DatastreamVerification
+  autoload :DatesMover
   autoload :Errors
   autoload :FileConfigurator
   autoload :Hooks
@@ -46,6 +29,7 @@ module FedoraMigrate
   autoload :RepositoryMigrator
   autoload :RightsMetadata
   autoload :RubydoraConnection
+  autoload :TargetConstructor
   autoload :TripleConverter
 
   class << self
@@ -62,7 +46,7 @@ module FedoraMigrate
 
     def source
       @source ||= FedoraMigrate::RubydoraConnection.new(fedora_config.credentials)
-    end 
+    end
 
     def find id
       FedoraMigrate.source.connection.find(id)
@@ -72,7 +56,6 @@ module FedoraMigrate
       migrator = FedoraMigrate::RepositoryMigrator.new(args[:namespace], args[:options])
       migrator.migrate_objects
       migrator.migrate_relationships
-      migrator.results
     end
 
   end

data/lib/fedora_migrate/content_mover.rb

@@ -0,0 +1,49 @@
+module FedoraMigrate
+  class ContentMover < Mover
+
+    def migrate
+      return nil_content_message if source.content.nil?
+      move_content
+      insert_date_created_by_application
+    end
+
+    def move_content
+      target.content = source.content
+      target.original_name = source.label.try(:gsub, /"/, '\"')
+      target.mime_type = source.mimeType
+      Logger.info "#{target.inspect}"
+      save
+    end
+
+    def insert_date_created_by_application
+      result = perform_sparql_insert
+      return true if result.status == 204
+      raise FedoraMigrate::Errors::MigrationError, "problem with sparql #{result.status} #{result.body}"
+    end
+
+    def sparql_insert
+<<-EOF
+PREFIX premis: <http://www.loc.gov/premis/rdf/v1#>
+PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
+DELETE WHERE { ?s premis:hasDateCreatedByApplication ?o } ;
+INSERT {
+  <> premis:hasDateCreatedByApplication "#{source.createDate.iso8601}"^^xsd:dateTime .
+}
+WHERE { }
+EOF
+    end
+
+    private
+
+    def nil_content_message
+      Logger.info "datastream '#{source.dsid}' is nil. It's probably defined in the target but not present in the source"
+      true
+    end
+
+    def perform_sparql_insert
+      ActiveFedora.fedora.connection.patch(target.metadata.metadata_uri, sparql_insert, "Content-Type" => "application/sparql-update")
+    end
+
+  end
+
+end

data/lib/fedora_migrate/datastream_mover.rb

@@ -1,6 +1,8 @@
 module FedoraMigrate
   class DatastreamMover < Mover
 
+    include DatastreamVerification
+
     attr_accessor :versionable
 
     def post_initialize
@@ -14,12 +16,20 @@ module FedoraMigrate
     def target_versionable?
       if target.respond_to?(:versionable?)
         target.versionable?
-      else 
+      else
         false
       end
     end
-    
+
     def migrate
+      before_datastream_migration
+      migrate_datastream
+      after_datastream_migration
+    end
+
+    private
+
+    def migrate_datastream
       if versionable?
         migrate_versions
       else
@@ -27,50 +37,25 @@ module FedoraMigrate
       end
     end
 
-    private
-
-    # Reloading the target, otherwise #get_checksum is nil
+    # Reload the target, otherwise the checksum is nil
     def migrate_current
       migrate_content
       target.reload
-      verify
+      valid?
     end
 
+    # Rubydora stores the versions array as the most recent first. We explicitly sort them according to createDate
     def migrate_versions
-      source.versions.each do |version|
+      source.versions.sort { |a,b| a.createDate <=> b.createDate }.each do |version|
         migrate_content(version)
-        target.create_version
-        verify(version)
+        target.create_version unless application_creates_versions?
+        valid?(version)
       end
     end
 
-    # TODO: lastModified isn't the right place for the original creation date (issue #1)
     def migrate_content datastream=nil
       datastream ||= source
-      if datastream.content.nil?
-        Logger.info "datastream '#{datastream.dsid}' is nil. It's probably defined in the target but not present in the source"
-        return true
-      end
-      target.content = datastream.content
-      target.original_name = datastream.label
-      target.mime_type = datastream.mimeType
-      target.last_modified = datastream.createDate
-      Logger.info "#{target.inspect}"
-      save
-    end
-
-    # TODO: Reporting mechanism? If there isn't a checksum it defaults to "none" (issue #4)
-    def verify datastream=nil
-      datastream ||= source
-      target_checksum = get_checksum
-      return true if datastream.checksum == "none"
-      unless datastream.checksum == target_checksum.split(/:/).last
-        Logger.fatal "expected #{datastream.dsid} #{datastream.checksumType} #{datastream.checksum} to match #{target_checksum}"
-      end
-    end
-
-    def get_checksum
-      target.digest.first.to_s
+      FedoraMigrate::ContentMover.new(datastream, target).migrate
     end
 
   end