hydra-file_characterization 0.3.1 → 1.1.1

data/Gemfile

@@ -2,3 +2,11 @@ source 'https://rubygems.org'
 
 # Specify your gem's dependencies in hydra/file_characterization.gemspec
 gemspec
+
+if ENV['RAILS_VERSION']
+  if ENV['RAILS_VERSION'] == 'edge'
+    gem 'rails', github: 'rails/rails'
+  else
+    gem 'rails', ENV['RAILS_VERSION']
+  end
+end

data/LICENSE

@@ -1,16 +1,14 @@
-##########################################################################
-#
-# Copyright 2013 Notre Dame
-# Additional copyright may be held by others, as reflected in the commit log
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+Copyright 2013 Notre Dame
+Additional copyright may be held by others, as reflected in the commit history. 
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -1,12 +1,28 @@
-# hydra-file_chracterization [![Version](https://badge.fury.io/rb/hydra-file_characterization.png)](http://badge.fury.io/rb/hydra-file_characterization) [![Build Status](https://travis-ci.org/projecthydra/hydra-file_characterization.png?branch=master)](https://travis-ci.org/projecthydra/hydra-file_characterization) [![Dependency Status](https://gemnasium.com/projecthydra/hydra-file_characterization.png)](https://gemnasium.com/projecthydra/hydra-file_characterization)
+# hydra-file_characterization
 
-Hydra::FileCharacterization as (extracted from Sufia and Hydra::Derivatives)
+Code: [![Version](https://badge.fury.io/rb/hydra-file_characterization.png)](http://badge.fury.io/rb/hydra-file_characterization) [![CircleCI](https://circleci.com/gh/samvera/hydra-file_characterization.svg?style=svg)](https://circleci.com/gh/samvera/hydra-file_characterization) [![Coverage Status](https://coveralls.io/repos/github/samvera/hydra-file_characterization/badge.svg?branch=master)](https://coveralls.io/github/samvera/hydra-file_characterization?branch=master)
 
-## Purpose
+Docs: [![Contribution Guidelines](http://img.shields.io/badge/CONTRIBUTING-Guidelines-blue.svg)](./CONTRIBUTING.md) [![Apache 2.0 License](http://img.shields.io/badge/APACHE2-license-blue.svg)](./LICENSE)
 
-To provide a wrapper for file characterization
+Jump in: [![Slack Status](http://slack.samvera.org/badge.svg)](http://slack.samvera.org/)
 
-## How To Use
+# What is hydra-file_characterization?
+
+Provides a wrapper for file characterization.
+
+## Product Owner & Maintenance
+
+hydra-file_characterization is a Core Component of the Samvera community. The documentation for what this means can be found [here](http://samvera.github.io/core_components.html#requirements-for-a-core-component).
+
+### Product Owner
+
+[little9](https://github.com/little9)
+
+# Help
+
+The Samvera community is here to help. Please see our [support guide](./SUPPORT.md).
+
+# Getting Started
 
 If you are using Rails add the following to an initializer (./config/initializers/hydra-file_characterization_config.rb):
 
@@ -16,12 +32,20 @@ Hydra::FileCharacterization.configure do |config|
 end
 ```
 
-You can call a single characterizer…
 ```ruby
-xml_string = Hydra::FileCharacterization.characterize(contents_of_a_file, 'file.rb', :fits)
+Hydra::FileCharacterization.characterize(File.read(filename), File.basename(filename), :fits)
 ```
 
-…for this particular call, you can specify custom fits path…
+- Why `file.read`? To highlight that we want a string. In the case of ActiveFedora, we have a StringIO instead of a file.
+- Why `file.basename`? In the case of Fits, the characterization takes cues from the extension name.
+
+You can call a single characterizer...
+
+```ruby
+xml_string = Hydra::FileCharacterization.characterize(File.read("/path/to/my/file.rb"), 'file.rb', :fits)
+```
+
+...for this particular call, you can specify custom fits path...
 
 ```ruby
 xml_string = Hydra::FileCharacterization.characterize(contents_of_a_file, 'file.rb', :fits) do |config|
@@ -29,7 +53,7 @@ xml_string = Hydra::FileCharacterization.characterize(contents_of_a_file, 'file.
 end
 ```
 
-…or even make the path callable…
+...or even make the path callable...
 
 ```ruby
 xml_string = Hydra::FileCharacterization.characterize(contents_of_a_file, 'file.rb', :fits) do |config|
@@ -37,7 +61,7 @@ xml_string = Hydra::FileCharacterization.characterize(contents_of_a_file, 'file.
 end
 ```
 
-…or even create your custom characterizer on the file…
+...or even create your custom characterizer on the file...
 
 ```ruby
 xml_string = Hydra::FileCharacterization.characterize(contents_of_a_file, 'file.rb', :my_characterizer) do |config|
@@ -51,9 +75,26 @@ You can also call multiple characterizers at the same time.
 fits_xml, ffprobe_xml = Hydra::FileCharacterization.characterize(contents_of_a_file, 'file.rb', :fits, :ffprobe)
 ```
 
-* Why `file.read`? To highlight that we want a string. In the case of ActiveFedora, we have a StringIO instead of a file.
-* Why `file.basename`? In the case of Fits, the characterization takes cues from the extension name.
-
 ## Registering New Characterizers
 
 This is possible by adding a characterizer to the `Hydra::FileCharacterization::Characterizers`' namespace.
+
+## Releasing
+
+1. `bundle install`
+2. Increase the version number in `lib/hydra/file_characterization/version.rb`
+3. Increase the same version number in `.github_changelog_generator`
+4. Update `CHANGELOG.md` by running this command:
+
+  ```
+  github_changelog_generator --user samvera --project hydra-file_characterization --token YOUR_GITHUB_TOKEN_HERE
+  ```
+
+5. Commit these changes to the master branch
+6. Run `rake release`
+
+# Acknowledgments
+
+This software has been developed by and is brought to you by the Samvera community. Learn more at the [Samvera website](http://samvera.org/).
+
+![Samvera Logo](https://wiki.duraspace.org/download/thumbnails/87459292/samvera-fall-font2-200w.png?version=1&modificationDate=1498550535816&api=v2)

data/Rakefile

@@ -1 +1,7 @@
 require "bundler/gem_tasks"
+require 'rspec/core'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/SUPPORT.md

@@ -0,0 +1,5 @@
+If you would like to report an issue, first search [the list of issues](https://github.com/samvera/hydra-file_characterization/issues/) to see if someone else has already reported it, and then feel free to [create a new issue](https://github.com/samvera/hydra-file_characterization/issues/new).
+i
+If you have questions or need help, please email [the Samvera community tech list](https://groups.google.com/forum/#!forum/samvera-tech) or stop by the #dev channel in [the Samvera community Slack team](https://wiki.duraspace.org/pages/viewpage.action?pageId=87460391#Getintouch!-Slack).
+
+You can learn more about the various Samvera communication channels on the [Get in touch!](https://wiki.duraspace.org/pages/viewpage.action?pageId=87460391) wiki page.

data/hydra-file_characterization.gemspec

@@ -29,7 +29,11 @@ Gem::Specification.new do |gem|
   gem.require_paths = ["lib"]
 
   gem.add_dependency "activesupport", ">= 3.0.0"
-  gem.add_development_dependency "rspec"
+  gem.add_development_dependency 'coveralls'
+  gem.add_development_dependency 'github_changelog_generator'
   gem.add_development_dependency "guard"
   gem.add_development_dependency 'guard-rspec'
+  gem.add_development_dependency "rake"
+  gem.add_development_dependency "rspec"
+  gem.add_development_dependency 'rspec_junit_formatter'
 end

data/lib/hydra/file_characterization/characterizer.rb

@@ -19,40 +19,60 @@ module Hydra::FileCharacterization
         raise Hydra::FileCharacterization::FileNotFoundError.new("File: #{filename} does not exist.")
       end
 
-      if tool_path.respond_to?(:call)
-        tool_path.call(filename)
-      else
-        internal_call
-      end
+      post_process(output)
     end
 
     def tool_path
       @tool_path || self.class.tool_path || convention_based_tool_name
     end
 
+    def logger
+      @logger ||= activefedora_logger || Logger.new(STDERR)
+    end
+
     protected
 
-    def convention_based_tool_name
-      self.class.name.split("::").last.downcase
-    end
+      # Override this method if you want your processor to mutate the
+      # raw output
+      def post_process(raw_output)
+        raw_output
+      end
 
-    def internal_call
-      stdin, stdout, stderr, wait_thr = popen3(command)
-      begin
-        out = stdout.read
-        err = stderr.read
-        exit_status = wait_thr.value
-        raise "Unable to execute command \"#{command}\"\n#{err}" unless exit_status.success?
-        out
-      ensure
-        stdin.close
-        stdout.close
-        stderr.close
+      def convention_based_tool_name
+        self.class.name.split("::").last.downcase
       end
-    end
 
-    def command
-      raise NotImplementedError, "Method #command should be overriden in child classes"
-    end
+      def internal_call
+        stdin, stdout, stderr, wait_thr = popen3(command)
+        begin
+          out = stdout.read
+          err = stderr.read
+          exit_status = wait_thr.value
+          raise "Unable to execute command \"#{command}\"\n#{err}" unless exit_status.success?
+          out
+        ensure
+          stdin.close
+          stdout.close
+          stderr.close
+        end
+      end
+
+      def command
+        raise NotImplementedError, "Method #command should be overriden in child classes"
+      end
+
+    private
+
+      def output
+        if tool_path.respond_to?(:call)
+          tool_path.call(filename)
+        else
+          internal_call
+        end
+      end
+
+      def activefedora_logger
+        ActiveFedora::Base.logger if defined? ActiveFedora
+      end
   end
 end

data/lib/hydra/file_characterization/characterizers.rb

@@ -29,3 +29,4 @@ end
 
 require 'hydra/file_characterization/characterizers/fits'
 require 'hydra/file_characterization/characterizers/ffprobe'
+require 'hydra/file_characterization/characterizers/fits_servlet'

data/lib/hydra/file_characterization/characterizers/fits.rb

@@ -1,12 +1,23 @@
 require 'hydra/file_characterization/exceptions'
 require 'hydra/file_characterization/characterizer'
+require 'logger'
 module Hydra::FileCharacterization::Characterizers
   class Fits < Hydra::FileCharacterization::Characterizer
 
     protected
-    def command
-      "#{tool_path} -i \"#{filename}\""
-    end
 
+      def command
+        "#{tool_path} -i \"#{filename}\""
+      end
+
+      # Remove any non-XML output that precedes the <?xml> tag
+      # See: https://github.com/harvard-lts/fits/issues/20
+      #      https://github.com/harvard-lts/fits/issues/40
+      #      https://github.com/harvard-lts/fits/issues/46
+      def post_process(raw_output)
+        md = /\A(.*)(<\?xml.*)\Z/m.match(raw_output)
+        logger.warn "FITS produced non-xml output: \"#{md[1].chomp}\"" unless md[1].empty?
+        md[2]
+      end
   end
 end

data/lib/hydra/file_characterization/characterizers/fits_servlet.rb

@@ -0,0 +1,23 @@
+require 'hydra/file_characterization/exceptions'
+require 'hydra/file_characterization/characterizer'
+require 'logger'
+module Hydra::FileCharacterization::Characterizers
+  class FitsServlet < Hydra::FileCharacterization::Characterizer
+
+    protected
+
+      def command
+        "curl -k -F datafile=@#{filename} #{ENV['FITS_SERVLET_URL']}/examine"
+      end
+
+      # Remove any non-XML output that precedes the <?xml> tag
+      # See: https://github.com/harvard-lts/fits/issues/20
+      #      https://github.com/harvard-lts/fits/issues/40
+      #      https://github.com/harvard-lts/fits/issues/46
+      def post_process(raw_output)
+        md = /\A(.*)(<\?xml.*)\Z/m.match(raw_output)
+        logger.warn "FITS produced non-xml output: \"#{md[1].chomp}\"" unless md[1].empty?
+        md[2]
+      end
+  end
+end

data/lib/hydra/file_characterization/version.rb

@@ -1,5 +1,5 @@
 module Hydra
   module FileCharacterization
-    VERSION = "0.3.1"
+    VERSION = "1.1.1"
   end
 end

data/spec/lib/hydra/file_characterization/characterizer_spec.rb

@@ -6,7 +6,8 @@ module Hydra::FileCharacterization
     let(:instance_tool_path) { nil }
     let(:class_tool_path) { nil }
 
-    subject { Hydra::FileCharacterization::Characterizer.new(filename, instance_tool_path) }
+    let(:characterizer) { Hydra::FileCharacterization::Characterizer.new(filename, instance_tool_path) }
+    subject { characterizer }
     around(:each) do |example|
       Hydra::FileCharacterization::Characterizer.tool_path = class_tool_path
       example.run
@@ -30,23 +31,25 @@ module Hydra::FileCharacterization
     end
 
     context 'tool_path' do
+      subject { characterizer.tool_path }
+
       context 'with custom instance tool_path' do
         let(:instance_tool_path) { '/arbitrary/path' }
         let(:class_tool_path) { '/a_different/path' }
 
-        its(:tool_path) { should eq instance_tool_path}
+        it { is_expected.to eq instance_tool_path }
       end
 
       context 'with custom class tool_path' do
         let(:instance_tool_path) { nil }
         let(:class_tool_path) { '/a_different/path' }
 
-        its(:tool_path) { should eq class_tool_path}
+        it { is_expected.to eq class_tool_path }
       end
 
       context 'without a specified tool_path' do
-        its(:tool_path) { should eq 'characterizer' }
+        it { is_expected.to eq 'characterizer' }
       end
     end
   end
-end
+end

data/spec/lib/hydra/file_characterization/characterizers/fit_servlet_spec.rb

@@ -0,0 +1,68 @@
+require 'spec_helper'
+require 'hydra/file_characterization/characterizers/fits_servlet'
+
+module Hydra::FileCharacterization::Characterizers
+  describe FitsServlet do
+    let(:fits) { Fits.new(filename) }
+
+    describe "#call", unless: ENV['TRAVIS'] do
+      subject { fits.call }
+
+      context 'validfile' do
+        let(:filename) { fixture_file('brendan_behan.jpeg') }
+        it { is_expected.to include(%(<identity format="JPEG File Interchange Format" mimetype="image/jpeg")) }
+      end
+
+      context 'invalidFile' do
+        let(:filename) { fixture_file('nofile.pdf') }
+        it "raises an error" do
+          expect { subject }.to raise_error(Hydra::FileCharacterization::FileNotFoundError)
+        end
+      end
+
+      context 'corruptFile' do
+        let(:filename) { fixture_file('brendan_broken.dxxd') }
+        it { is_expected.to include(%(<identity format="Unknown Binary" mimetype="application/octet-stream")) }
+      end
+
+      context 'zip file should be characterized not its contents' do
+        let(:filename) { fixture_file('archive.zip') }
+        it { is_expected.to include(%(<identity format="ZIP Format" mimetype="application/zip"))}
+      end
+    end
+
+    context 'when JHOVE adds non-xml' do
+      # https://github.com/harvard-lts/fits/issues/20
+      subject { fits.call }
+
+      before do
+        expect(fits.logger).to receive(:warn)
+        allow(fits).to receive(:internal_call).and_return(
+          'READBOX seen=true
+<?xml version="1.0" encoding="UTF-8"?>
+<fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://hul.harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.8.2" timestamp="15/09/14 10:00 AM">
+<identification/></fits>')
+      end
+
+      let(:filename) { fixture_file('brendan_behan.jpeg') }
+      it { is_expected.not_to include('READBOX') }
+    end
+
+    context "when FITS itself adds non-xml" do
+      # https://github.com/harvard-lts/fits/issues/46
+      subject { fits.call }
+
+      before do
+        expect(fits.logger).to receive(:warn)
+        allow(fits).to receive(:internal_call).and_return(
+          '2015-10-15 17:14:25,761 ERROR [main] ToolBelt:79 - Thread 1 error initializing edu.harvard.hul.ois.fits.tools.droid.Droid: edu.harvard.hul.ois.fits.exceptions.FitsToolException  Message: DROID cannot run under Java 8
+<?xml version="1.0" encoding="UTF-8"?>
+<fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://hul.harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.8.2" timestamp="15/09/14 10:00 AM">
+<identification/></fits>')
+      end
+
+      let(:filename) { fixture_file('brendan_behan.jpeg') }
+      it { is_expected.not_to include('FitsToolException') }
+    end
+  end
+end

data/spec/lib/hydra/file_characterization/characterizers/fits_spec.rb

@@ -2,37 +2,67 @@ require 'spec_helper'
 require 'hydra/file_characterization/characterizers/fits'
 
 module Hydra::FileCharacterization::Characterizers
-
   describe Fits do
+    let(:fits) { Fits.new(filename) }
 
-    subject { Fits.new(filename) }
+    describe "#call", unless: ENV['TRAVIS'] do
+      subject { fits.call }
 
-    describe 'validfile' do
-      let(:filename) { fixture_file('brendan_behan.jpeg') }
-      it '#call' do
-        expect(subject.call).to include(%(<identity format="JPEG File Interchange Format" mimetype="image/jpeg"))
+      context 'validfile' do
+        let(:filename) { fixture_file('brendan_behan.jpeg') }
+        it { is_expected.to include(%(<identity format="JPEG File Interchange Format" mimetype="image/jpeg")) }
       end
-    end
 
-    describe 'invalidFile' do
-      let(:filename) { fixture_file('nofile.pdf') }
-      it "should raise an error if the path does not contain the file" do
-        expect {subject.call}.to raise_error(Hydra::FileCharacterization::FileNotFoundError)
+      context 'invalidFile' do
+        let(:filename) { fixture_file('nofile.pdf') }
+        it "raises an error" do
+          expect { subject }.to raise_error(Hydra::FileCharacterization::FileNotFoundError)
+        end
       end
-    end
 
-    describe 'corruptFile' do
-      let(:filename) { fixture_file('brendan_broken.dxxd') }
-      it "should return xml showing Unknown Binary and application/octet-stream mimetype" do
-        expect(subject.call).to include(%(<identity format="Unknown Binary" mimetype="application/octet-stream"))
+      context 'corruptFile' do
+        let(:filename) { fixture_file('brendan_broken.dxxd') }
+        it { is_expected.to include(%(<identity format="Unknown Binary" mimetype="application/octet-stream")) }
+      end
+
+      context 'zip file should be characterized not its contents' do
+        let(:filename) { fixture_file('archive.zip') }
+        it { is_expected.to include(%(<identity format="ZIP Format" mimetype="application/zip"))}
       end
     end
 
-    describe 'zip file should be characterized not its contents' do
-      let(:filename) { fixture_file('archive.zip') }
-      its(:call) { should include(%(<identity format="ZIP Format" mimetype="application/zip"))}
+    context 'when JHOVE adds non-xml' do
+      # https://github.com/harvard-lts/fits/issues/20
+      subject { fits.call }
+
+      before do
+        expect(fits.logger).to receive(:warn)
+        allow(fits).to receive(:internal_call).and_return(
+          'READBOX seen=true
+<?xml version="1.0" encoding="UTF-8"?>
+<fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://hul.harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.8.2" timestamp="15/09/14 10:00 AM">
+<identification/></fits>')
+      end
+
+      let(:filename) { fixture_file('brendan_behan.jpeg') }
+      it { is_expected.not_to include('READBOX') }
     end
 
-  end
+    context "when FITS itself adds non-xml" do
+      # https://github.com/harvard-lts/fits/issues/46
+      subject { fits.call }
+
+      before do
+        expect(fits.logger).to receive(:warn)
+        allow(fits).to receive(:internal_call).and_return(
+          '2015-10-15 17:14:25,761 ERROR [main] ToolBelt:79 - Thread 1 error initializing edu.harvard.hul.ois.fits.tools.droid.Droid: edu.harvard.hul.ois.fits.exceptions.FitsToolException  Message: DROID cannot run under Java 8
+<?xml version="1.0" encoding="UTF-8"?>
+<fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://hul.harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.8.2" timestamp="15/09/14 10:00 AM">
+<identification/></fits>')
+      end
 
+      let(:filename) { fixture_file('brendan_behan.jpeg') }
+      it { is_expected.not_to include('FitsToolException') }
+    end
+  end
 end