assembly-objectfile 1.5.0

data/.gitignore

@@ -0,0 +1,13 @@
+.rvmrc
+*.gem
+.DS_Store
+.bundle
+Gemfile.lock
+config/certs/*
+config/environments/*
+doc/*
+log/*
+pkg/*
+tags
+tmp
+.yardoc/*

data/.rspec

@@ -0,0 +1 @@
+-r spec/spec_helper.rb

data/.rvmrc.example

@@ -0,0 +1 @@
+rvm use 1.8.7@assembly-objectfile --create

data/Gemfile

@@ -0,0 +1,5 @@
+source "http://rubygems.org"
+source "http://sul-gems.stanford.edu"
+
+# Specify your gem's dependencies in .gemspec
+gemspec

data/README.rdoc

@@ -0,0 +1,94 @@
+= Assembly-ObjectFile Gem
+
+==Overview 
+This gem contains classes used by the Stanford University Digital Library to perform
+file operations necessary for accessioning of content.  It is also used by related gems to perform
+content type specific operations (such as jp2 generation).
+
+==Releases
+
+- 1.0.0 initial release
+- 1.0.1 add in a valid_image? method
+- 1.0.2 add new mimetype and encoding methods
+- 1.0.3 add new object_type method
+- 1.0.4 add additional configuration parameters that are used by assembly and other consumers of the gem
+- 1.0.5 try and get mimetype from exif before resorting to unix level file command
+- 1.1.0 add methods to compute sha1 and md5 for object using checksum-tools gem
+- 1.1.1 change computation of mimetype and encoding to work in more situations
+- 1.1.2 change jp2able? and image? methods so they return a file not found error if the file is not supplied, added a method to indicate if a file is found
+- 1.1.3 prepare for release listing on DLSS release board
+- 1.1.4 valid_image? will now return true for jp2 mimetypes
+- 1.1.5 valid_image? should be false if mimetype is correct but profile is not found
+- 1.1.6 add jp2able? method
+- 1.1.7 change the behavior of jp2able? and valid_image?
+- 1.1.8 update how the version number is set to allow users to see the gem version # more easily
+- 1.1.9 switch jpeg mimetype temporarily to publish=no, preserve=yes
+- 1.2.0 move content metadata generation method to this gem from assembly-image
+- 1.2.1 allow content metadata to add user supplied checksums
+- 1.2.2 allow content metadata to bundle files into resources by filename or DPG filename specification, add book_as_image and file style content metadata generation
+- 1.2.3 small change to a parameter passed to content metadata generation
+- 1.2.4 allow user to control how file ID paths are generated in content metadata by using a 'relative_path' attribute
+- 1.2.5 add a class method to find the common directory between an array of filenames passed in
+- 1.2.6 bug fix in content metadata generation, and allow book types to have single <file> resource nodes if they do not contain any images
+- 1.2.8 remove dependency on ChecksumTools gem to make it Ruby 1.9 compatible
+- 1.2.9 automatically strip druid: prefix in content metadata generation method
+- 1.2.10 bug fix
+- 1.2.11 add ability to suppress <xml> tag from generated contentMetadata
+- 1.3.0 continued refinement of content metadata generation for objects with download
+- 1.3.1 allow the user specify a label with the ObjectFile object that is picked up when generating content metadata (in specifying resource labels)
+- 1.3.2 allow the user to set the label on object creation
+- 1.3.3 update tests to avoid dependency on kakadu; contentMetadata will now generate new resources of type=object when files are present in special DPG folders
+- 1.3.4 fix rspec test to have it run on CI server
+- 1.3.5 add a parameter to flatten folder structure when creating file IDs; increment resource labels by object type
+- 1.3.6 allow user to supply default file attributes as well as by mimetype --- useful if file attributes should be added and are the same regardless of mimetype
+- 1.3.7 add a new bundle style called "prebundled" which allows users to pass in an array of arrays
+- 1.3.8 update to latest lyberteam devel gems
+- 1.3.9 compute md5 and sha1 separately when needed
+- 1.4.0 compute mimetype correctly even if exif information in a file is damaged
+- 1.4.1 fix errors that could error if there was a space in the filename
+- 1.4.2 Support map style content metadata; don't compute mimetype when generating content metadata unless its needed
+- 1.4.3 object_type method should return :other if it is an unknown mimetype
+- 1.4.4 produce blank content metadata if no objects are passed in 
+- 1.4.5 allow the user to supply optional file specific attributes for publish, preserve, shelve for creating content metadata
+- 1.4.6 add dirname attribute to objectfile class
+- 1.4.7 add an additional default mimetype for file perservation attributes
+- 1.4.8 compute mimetype with file unix command by default, and then check for existence of mimetype in exif unless we have a "trusted" mimetype
+- 1.4.9 update list of trusted mimetypes
+- 1.5.0 add the ability to skip auto label generation for resources if no labels are supplied
+
+==Usage
+
+The gem currently has methods for:
+- filesize
+- exif
+- generate content metadata
+
+==Running tests
+
+	bundle exec rspec spec
+
+==Generate documentation
+To generate documentation into the "doc" folder:
+
+	yard
+
+To keep a local server running with up to date code documentation that you can view in your browser:
+
+	yard server --reload
+
+==Prerequisites
+
+1. Exiftool
+            
+   RHEL: (RPM to install comming soon)
+   Download latest version from:  http://www.sno.phy.queensu.ca/~phil/exiftool
+
+       tar -xf Image-ExifTool-#.##.tar.gz
+       cd Image-ExifTool-#.##
+       perl Makefile.PL
+       make test
+       sudo make install
+
+   Mac users:
+       brew install exiftool
+

data/Rakefile

@@ -0,0 +1,4 @@
+require "bundler/gem_tasks"
+
+require 'dlss/rake/dlss_release'
+Dlss::Release.new

data/assembly-objectfile.gemspec

@@ -0,0 +1,29 @@
+$LOAD_PATH.push File.expand_path("../lib", __FILE__)
+require 'assembly-objectfile/version'
+
+Gem::Specification.new do |s|
+  s.name        = 'assembly-objectfile'
+  s.version     = Assembly::ObjectFile::VERSION
+  s.authors     = ["Peter Mangiafico", "Renzo Sanchez-Silva","Monty Hindman","Tony Calavano"]
+  s.email       = ["pmangiafico@stanford.edu"]
+  s.homepage    = ""
+  s.summary     = %q{Ruby immplementation of file services needed to prepare objects to be accessioned in SULAIR digital library}
+  s.description = %q{Get exif data, file sizes and more.}
+
+  s.rubyforge_project = 'assembly-objectfile'
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ['lib']
+
+  s.add_dependency 'mini_exiftool'
+  s.add_dependency 'mime-types'
+  s.add_dependency 'nokogiri'
+
+  s.add_development_dependency "rspec", "~> 2.6"
+  s.add_development_dependency "lyberteam-devel", '>= 1.0.1'
+  s.add_development_dependency "lyberteam-gems-devel", "> 1.0.0"
+  s.add_development_dependency "yard"
+  
+end

data/bin/console

@@ -0,0 +1,8 @@
+#! /usr/bin/env ruby
+
+require 'rubygems'
+require 'irb'
+
+require File.expand_path(File.dirname(__FILE__) + '/../config/boot')
+
+IRB.start

data/bin/run_all_tests

@@ -0,0 +1,3 @@
+#! /bin/bash
+
+bundle exec rspec spec

data/config/boot.rb

@@ -0,0 +1,9 @@
+require 'rubygems'
+
+environment  = ENV['ENVIRONMENT'] ||= 'development'
+project_root = File.expand_path(File.dirname(__FILE__) + '/..')
+
+# Load config for current environment.
+$LOAD_PATH.unshift(project_root + '/lib')
+
+require 'assembly-objectfile'

data/lib/assembly-objectfile.rb

@@ -0,0 +1,35 @@
+module Assembly
+  
+  # the path to the gem, used to access profiles stored with the gem
+  PATH_TO_GEM = File.expand_path(File.dirname(__FILE__) + '/..')
+
+  # if input image is not one of these mime types, it will not be regarded as a valid image
+  VALID_IMAGE_MIMETYPES=["image/jpeg","image/tiff","image/jp2","image/tif"]
+  
+  # the list of mimetypes that will be "trusted" by the unix file command; if a mimetype other than one of these is returned
+  #  by the file command, then a check will be made to see if exif data exists...if so, the mimetype returned by the exif data will be used
+  #  if no exif data exists, then the mimetype returned by the unix file command will be used
+  TRUSTED_MIMETYPES=["text/plain","plain/text","application/pdf","text/html","application/xml"]
+  
+  # default publish/preserve/shelve attributes used in content metadata
+  FILE_ATTRIBUTES=Hash.new
+  # if no mimetype specific attributes are specified for a given file, define some defaults, and override for specific mimetypes below
+  FILE_ATTRIBUTES['default']={:preserve=>'yes',:shelve=>'no',:publish=>'no'}
+  FILE_ATTRIBUTES['image/tif']={:preserve=>'yes',:shelve=>'no',:publish=>'no'}
+  FILE_ATTRIBUTES['image/tiff']={:preserve=>'yes',:shelve=>'no',:publish=>'no'}
+  FILE_ATTRIBUTES['image/jp2']={:preserve=>'no',:shelve=>'yes',:publish=>'yes'}
+  FILE_ATTRIBUTES['image/jpeg']={:preserve=>'yes',:shelve=>'no',:publish=>'no'}
+  FILE_ATTRIBUTES['audio/wav']={:preserve=>'yes',:shelve=>'no',:publish=>'no'}
+  FILE_ATTRIBUTES['audio/x-wav']={:preserve=>'yes',:shelve=>'no',:publish=>'no'}
+  FILE_ATTRIBUTES['audio/mp3']={:preserve=>'no',:shelve=>'yes',:publish=>'yes'}
+  FILE_ATTRIBUTES['audio/mpeg']={:preserve=>'no',:shelve=>'yes',:publish=>'yes'}
+  FILE_ATTRIBUTES['application/pdf']={:preserve=>'yes',:shelve=>'yes',:publish=>'yes'}
+  FILE_ATTRIBUTES['plain/text']={:preserve=>'yes',:shelve=>'yes',:publish=>'yes'}
+  FILE_ATTRIBUTES['text/plain']={:preserve=>'yes',:shelve=>'yes',:publish=>'yes'}
+  
+end
+
+require 'assembly-objectfile/content_metadata'
+require 'assembly-objectfile/object_fileable'
+require 'assembly-objectfile/object_file'
+require 'assembly-objectfile/version'

data/lib/assembly-objectfile/content_metadata.rb

@@ -0,0 +1,229 @@
+require 'nokogiri'
+
+module Assembly
+
+  SPECIAL_DPG_FOLDERS=['31','50']  # these special dpg folders will force any files contained in them into their own resources, regardless of filenaming convention
+                                   # these are used when :bundle=>:dpg only
+                                   
+  # This class generates content metadata for image files
+  class ContentMetadata
+    
+      # Generates image content XML metadata for a repository object.
+      # This method only produces content metadata for images
+      # and does not depend on a specific folder structure.  Note that it is class level method.
+      #
+      # @param [Hash] params a hash containg parameters needed to produce content metadata
+      #   :druid = required - a string of druid of the repository object's druid id (with or without 'druid:' prefix)
+      #   :objects = required - an array of Assembly::ObjectFile objects containing the list of files to add to content metadata
+      #                NOTE: if you set the :bundle option to "prebundlded", you will need to pass in an array of arrays, and not a flat array, as noted below      
+      #   :style = optional - a symbol containing the style of metadata to create, allowed values are
+      #                 :simple_image (default), contentMetadata type="image", resource type="image"
+      #                 :file, contentMetadata type="file", resource type="file"      
+      #                 :simple_book, contentMetadata type="book", resource type="page", but any resource which has file(s) other than an image, and also contains no images at all, will be resource type="object"
+      #                 :book_with_pdf, contentMetadata type="book", resource type="page", but any resource which has any file(s) other than an image will be resource type="object"
+      #                 :book_as_image, as simple_book, but with contentMetadata type="book", resource type="image" (same rule applies for resources with non images)
+      #                 :map, like simple_image, but with contentMetadata type="map", resoruce type="image" 
+      #   :bundle = optional - a symbol containing the method of bundling files into resources, allowed values are
+      #                 :default = all files get their own resources (default)
+      #                 :filename = files with the same filename but different extensions get bundled together in a single resource
+      #                 :dpg = files representing the same image but of different mimetype that use the SULAIR DPG filenaming standard (00 vs 05) get bundled together in a single resource
+      #                 :prebundlded = this option requires you to prebundled the files passed in as an array of arrays, indicating how files are bundlded into resources; this is the most flexible option since it gives you full control
+      #   :add_exif = optional - a boolean to indicate if exif data should be added (mimetype, filesize, image height/width, etc.) to each file, defaults to false and is not required if project goes through assembly
+      #   :add_file_attributes = optional - a boolean to indicate if publish/preserve/shelve attributes should be added using defaults or by supplied override by mime/type, defaults to false and is not required if project goes through assembly
+      #   :file_attributes = optional - a hash of file attributes by mimetype to use instead of defaults, only used if add_file_attributes is also true, 
+      #             If a mimetype match is not found in your hash, the default is used (either your supplied default or the gems).
+      #             e.g. {'default'=>{:preserve=>'yes',:shelve=>'yes',:publish=>'yes'},'image/tif'=>{:preserve=>'yes',:shelve=>'no',:publish=>'no'},'application/pdf'=>{:preserve=>'yes',:shelve=>'yes',:publish=>'yes'}}
+      #   :include_root_xml = optional - a boolean to indicate if the contentMetadata returned includes a root <?xml version="1.0"?> tag, defaults to true
+      #   :preserve_common_paths = optional - When creating the file "id" attribute, content metadata uses the "relative_path" attribute of the ObjectFile objects passed in.  If the "relative_path" attribute is not set,  the "path" attribute is used instead,
+      #                   which includes a full path to the file. If the "preserve_common_paths" parameter is set to false or left off, then the common paths of all of the ObjectFile's passed in are removed from any "path" attributes.  This should turn full paths into
+      #                   the relative paths that are required in content metadata file id nodes.  If you do not want this behavior, set "preserve_common_paths" to true.  The default it false.
+      #   :flatten_folder_structure = optional - Will remove *all* folder structure when genearting file IDs (e.g. DPG subfolders like '00','05' will be removed) when generating file IDs.  This is useful if the folder structure is flattened when staging files (like for DPG).  
+      #                                             The default is false.  If set to true, will override the "preserve_common_paths" parameter.  
+      #   :auto_labels = optional - Will add automated resource labels (e.g. "File 1") when labels are not provided by the user.  The default is true. 
+      # Example:
+      #    Assembly::Image.create_content_metadata(:druid=>'druid:nx288wh8889',:style=>:simple_image,:objects=>object_files,:add_file_attributes=>false)
+      def self.create_content_metadata(params={})
+
+        druid=params[:druid]
+        objects=params[:objects]
+
+        raise "No objects and/or druid supplied" if druid.nil? || objects.nil?
+        
+        pid=druid.gsub('druid:','') # remove druid prefix when creating IDs
+        
+        style=params[:style] || :simple_image
+        bundle=params[:bundle] || :default
+        add_exif=params[:add_exif] || false
+        auto_labels=(params[:auto_labels].nil? ? true : params[:auto_labels])
+        add_file_attributes=params[:add_file_attributes] || false
+        file_attributes=params[:file_attributes] || {}
+        preserve_common_paths=params[:preserve_common_paths] || false
+        flatten_folder_structure=params[:flatten_folder_structure] || false
+        include_root_xml=params[:include_root_xml]
+                      
+        all_paths=[]
+        objects.flatten.each do |obj| 
+          raise "File '#{obj.path}' not found" unless obj.file_exists?
+          all_paths << obj.path unless preserve_common_paths # collect all of the filenames into an array
+        end
+        
+        common_path=Assembly::ObjectFile.common_path(all_paths) unless preserve_common_paths # find common paths to all files provided if needed
+        
+        # these are the valid strings for each type of document to be use contentMetadata type and resourceType
+        content_type_descriptions={:file=>'file',:image=>'image',:book=>'book',:map=>'map'}
+        resource_type_descriptions={:object=>'object',:file=>'file',:image=>'image',:book=>'page',:map=>'image'}
+      
+        # global sequence for resource IDs
+        sequence = 0
+        
+        # a counter to use when creating auto-labels for resources, with incremenets for each type
+        resource_type_counters=Hash.new(0)
+        
+        # set the content type id
+        case style
+          when :simple_image
+            content_type_description = content_type_descriptions[:image]
+          when :file
+            content_type_description = content_type_descriptions[:file]
+          when :simple_book,:book_with_pdf
+            content_type_description = content_type_descriptions[:book]
+          when :book_as_image
+            content_type_description = content_type_descriptions[:book]
+          when :map
+            content_type_description = content_type_descriptions[:map]
+          else
+            raise "Supplied style not valid"
+        end
+          
+        
+        # determine how many resources to create
+        # setup an array of arrays, where the first array is the number of resources, and the second array is the object files containined in that resource
+        case bundle
+          when :default # one resource per object
+            resources=objects.collect {|obj| [obj]}
+          when :filename # one resource per distinct filename (excluding extension)
+            # loop over distinct filenames, this determines how many resources we will have and
+            # create one resource node per distinct filename, collecting the relevant objects with the distinct filename into that resource
+            resources=[]
+            distinct_filenames=objects.collect {|obj| obj.filename_without_ext}.uniq # find all the unique filenames in the set of objects, leaving off extensions and base paths
+            distinct_filenames.each {|distinct_filename| resources << objects.collect {|obj| obj if obj.filename_without_ext == distinct_filename}.compact }
+          when :dpg # group by DPG filename
+            # loop over distinct dpg base names, this determines how many resources we will have and
+            # create one resource node per distinct dpg base name, collecting the relevant objects with the distinct names into that resource
+            resources=[]
+            distinct_filenames=objects.collect {|obj| obj.dpg_basename}.uniq # find all the unique DPG filenames in the set of objects
+            distinct_filenames.each do |distinct_filename| 
+              resources << objects.collect {|obj| obj if obj.dpg_basename == distinct_filename && !self.is_special_dpg_folder?(obj.dpg_folder)}.compact
+            end
+            objects.each {|obj| resources << [obj] if self.is_special_dpg_folder?(obj.dpg_folder)} # certain subfolders require individual resources for files within them regardless of file-naming convention      
+          when :prebundled
+            # if the user specifies this method, they will pass in an array of arrays, indicating resources, so we don't need to bundle in the gem
+            resources=objects
+          else
+            raise "Invalid bundle method"
+        end
+        
+        resources.delete([]) # delete any empty elements
+        
+        builder = Nokogiri::XML::Builder.new do |xml|
+          xml.contentMetadata(:objectId => "#{pid}",:type => content_type_description) {
+            resources.each do |resource_files| # iterate over all the resources
+
+              # start a new resource element              
+              sequence += 1
+              resource_id = "#{pid}_#{sequence}"
+
+              # grab all of the file types within a resource into an array so we can decide what the resource type should be
+              resource_file_types=resource_files.collect {|obj| obj.object_type}
+              resource_has_non_images=((resource_file_types-[:image]).size > 0)
+              resource_from_special_dpg_folder=resource_files.collect {|obj| self.is_special_dpg_folder?(obj.dpg_folder)}.uniq
+                
+              if bundle == :dpg && resource_from_special_dpg_folder.include?(true)  # objects in the special DPG folders are always type=object when we using :bundle=>:dpg
+                resource_type_description = resource_type_descriptions[:object]
+              else # otherwise look at the style to determine the resource_type_description
+                case style
+                   when :simple_image 
+                     resource_type_description = resource_type_descriptions[:image]
+                   when :file
+                     resource_type_description = resource_type_descriptions[:file]           
+                   when :simple_book # in a simple book project, all resources are pages unless they are *all* non-images -- if so, switch the type to object
+                     resource_type_description = (resource_has_non_images && resource_file_types.include?(:image) == false) ? resource_type_descriptions[:object] : resource_type_descriptions[:book]
+                   when :book_as_image # same as simple book, but all resources are images instead of pages, unless we need to switch them to object type
+                     resource_type_description = (resource_has_non_images && resource_file_types.include?(:image) == false) ? resource_type_descriptions[:object] : resource_type_descriptions[:image]
+                   when :book_with_pdf # in book with PDF type, if we find a resource with *any* non images, switch it's type from book to object
+                     resource_type_description = resource_has_non_images ? resource_type_descriptions[:object] : resource_type_descriptions[:book]
+                   when :map # 
+                     resource_type_description = resource_type_descriptions[:map]
+                 end             
+              end
+              
+              resource_type_counters[resource_type_description.to_sym]+=1  # each resource type description gets its own incrementing counter
+                              
+              xml.resource(:id => resource_id,:sequence => sequence,:type => resource_type_description) {
+
+                # create a generic resource label if needed
+                resource_label = (auto_labels == true ? "#{resource_type_description.capitalize} #{resource_type_counters[resource_type_description.to_sym]}" : "") 
+                
+                # but if one of the files has a label, use it instead
+                resource_files.each {|obj| resource_label = obj.label unless obj.label.nil? || obj.label.empty? }
+                
+                xml.label(resource_label) unless resource_label.empty?
+
+                resource_files.each do |obj| # iterate over all the files in a resource
+              
+                  mimetype = obj.mimetype if (add_file_attributes || add_exif) # we only need to compute the mimetype if we are adding file attributes or exif info, otherwise skip it for performance reasons
+                
+                  # set file id attribute, first check the relative_path parameter on the object, and if it is set, just use that
+                  if obj.relative_path 
+                    file_id=obj.relative_path                      
+                  else 
+                    # if the relative_path attribute is not set, then use the path attribute and check to see if we need to remove the common part of the path 
+                    file_id=preserve_common_paths ? obj.path : obj.path.gsub(common_path,'')
+                    file_id=File.basename(file_id) if flatten_folder_structure
+                  end
+
+                  xml_file_params = {:id=> file_id}
+              
+                  if add_file_attributes
+                    file_attributes_hash=obj.file_attributes || file_attributes[mimetype] || file_attributes['default'] || Assembly::FILE_ATTRIBUTES[mimetype] || Assembly::FILE_ATTRIBUTES['default']
+                    xml_file_params.merge!({
+                      :preserve => file_attributes_hash[:preserve],
+                      :publish  => file_attributes_hash[:publish],
+                      :shelve   => file_attributes_hash[:shelve],
+                    })
+                  end
+                                                      
+                  xml_file_params.merge!({:mimetype => mimetype,:size => obj.filesize}) if add_exif
+                  xml.file(xml_file_params) {
+                    if add_exif # add exif info if the user requested it
+                      xml.checksum(obj.sha1, :type => 'sha1')
+                      xml.checksum(obj.md5, :type => 'md5')                                
+                      xml.imageData(:height => obj.exif.imageheight, :width => obj.exif.imagewidth) if obj.image? # add image data for an image
+                    elsif obj.provider_md5 || obj.provider_sha1 # if we did not add exif info, see if there are user supplied checksums to add
+                      xml.checksum(obj.provider_sha1, :type => 'sha1') if obj.provider_sha1
+                      xml.checksum(obj.provider_md5, :type => 'md5') if obj.provider_md5                                                    
+                    end #add_exif
+                  }
+                end # end resource_files.each
+              }
+            end # resources.each
+          }
+        end
+        
+        if include_root_xml == false
+          result = builder.doc.root.to_xml
+        else
+          result = builder.to_xml
+        end
+                
+        return result
+        
+      end # create_content_metadata
+      
+      def self.is_special_dpg_folder?(folder)
+        SPECIAL_DPG_FOLDERS.include?(folder)
+      end
+      
+  end # class
+  
+end # module