hydra-derivatives 3.2.2 → 3.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 505c538dd6ded3a65f826eb0d93040bc4edf2a1e
-  data.tar.gz: 0323253a69f02a91b84e07f546e42debd822ea7f
+  metadata.gz: b300c88a53fe0b8e79f9ace11145653e878f8614
+  data.tar.gz: dfe48e84c0bd64e74d77d3a4c9cb103f2aaee0dc
 SHA512:
-  metadata.gz: c27070a785686b16bb37c60d0a30cf175da384e2818942efb3d7994ff0eacf2a61b84aa0f1189216d4242ed235bda6bda1a43001b3dcb7ad09efcb5a9a085254
-  data.tar.gz: 453cec1e1d0b1f5940ad251f6271771e0f062f37e53342f80e31f783005b1a87615868d615e569389dcef1635e2332ef59e7a7f92048c93018e0a6e4f0876349
+  metadata.gz: 2f1f0183aae8af14487552ebf3c5f9453db4fa64c7f51fa9f1ac6bc4453cf58d3b9bf0a25c5296450c4006eb7b45b4fb6d6093f1cf46a7548d0a189854cfdb19
+  data.tar.gz: '058e7057eae73e19451e8d9926576305f85416bd1c3db0bdba279acc97bd629fa27d9ac6200dd56f84505a780d1371937e74f8eab970983a3276af1d3092fd27'

data/.travis.yml

@@ -6,8 +6,8 @@ before_install:
   - gem install bundler
 rvm:
   - 2.2.5
-  - 2.3.1
-  - 2.4.0
+  - 2.3.4
+  - 2.4.1
 env:
   global:
     - NOKOGIRI_USE_SYSTEM_LIBRARIES=true

data/README.md

@@ -52,6 +52,8 @@ Then when you call `obj.create_derivatives` two new files, 'thumbnail' and 'cont
 
 We recommend you run `obj.create_derivatives` in a background worker, because some derivative creation (especially videos) can take a long time.
 
+The ActiveEncode runner provides support for using external video transcoding services like Amazon Elastic Transcoder.  See [ActiveEncode](https://github.com/samvera-labs/active_encode) for details on specific adapters supported.
+
 ## Configuration
 
 ### Retrieving from a basic container in Fedora
@@ -74,6 +76,7 @@ Hydra::Derivatives::Processors::Video::Processor.timeout  = 10.minutes
 Hydra::Derivatives::Processors::Document.timeout = 5.minutes
 Hydra::Derivatives::Processors::Audio.timeout = 10.minutes
 Hydra::Derivatives::Processors::Image.timeout = 5.minutes
+Hydra::Derivatives::Processors::ActiveEncode.timeout = 5.minutes
 
 ```
 
@@ -88,6 +91,28 @@ Hydra::Derivatives::Processors::Video::Processor.config.mkv.codec = '-vcodec ffv
 Hydra::Derivatives::Processors::Video::Processor.config.jpeg.codec = '-vcodec mjpeg'
 ```
 
+### Configuration for Audio/Video Processing with ActiveEncode
+
+```ruby
+# Set the transcoding engine
+ActiveEncode::Base.engine_adapter = :elastic_transcoder
+
+# Sleep time (in seconds) to poll for status of encoding job
+Hydra::Derivatives.active_encode_poll_time = 10
+
+# If you want to use a different class for the source file service
+Hydra::Derivatives::ActiveEncodeDerivatives.source_file_service = MyCustomSourceFileService
+
+# If you want to use a different class for the output file service
+Hydra::Derivatives::ActiveEncodeDerivatives.output_file_service = MyCustomOutputFileService
+```
+
+Note: Please don't confuse these methods with the similar methods in the parent class:
+`Hydra::Derivatives.source_file_service` and `Hydra::Derivatives.output_file_service`
+
+For additional documentation on using ActiveEncode, see:
+* [Using Amazon Elastic Transcoder](doc/amazon_elastic_transcoder.md)
+
 ### Additional Directives
 
 #### Layers

data/VERSION

@@ -1 +1 @@
-3.2.2
+3.3.0

data/doc/amazon_elastic_transcoder.md

@@ -0,0 +1,116 @@
+# Create Audio and Video Derivatives using Amazon Elastic Transcoder
+
+`hydra-derivatives` uses the
+[active\_encode gem](https://github.com/projecthydra-labs/active_encode)
+to allow you to use different encoding services.
+These instructions are for Amazon's Elastic Transcoder service.
+
+## Prerequsites
+
+### Set up the Elastic Transcoder Pipeline
+
+Set up a pipeline on AWS Elastic Transcoder that defines:
+
+* input bucket
+* bucket for transcoded files
+* bucket for thumbnails
+
+### Configure AWS credentials
+
+Optional: If you don't want to pass these values in your ruby code using `Aws.config`, you can set environment variables instead:
+
+* AWS\_ACCESS\_KEY\_ID
+* AWS\_SECRET\_ACCESS\_KEY
+* AWS\_REGION
+
+### Install gems
+
+Add to your `Gemfile`:
+
+* aws-sdk
+
+### Configure initializer
+
+In an initializer file such as `config/initializers/active_encode.rb`, make sure you have the following code:
+
+```ruby
+# Use Amazon's Elastic Transcoder
+ActiveEncode::Base.engine_adapter = :elastic_transcoder
+```
+
+## How to create derivatives (Multiple derivatives per Elastic Transcoder job)
+
+```ruby
+# Access config for AWS
+Aws.config[:access_key_id] = 'put your access key here'
+Aws.config[:secret_access_key] = 'put your secret key here'
+Aws.config[:region] = 'us-east-1'
+
+# The pipeline that I set up in Elastic Transcoder
+pipeline_id = '1490715200916-25b08y'
+
+# The file "sample_data.mp4" has already been uploaded to the input bucket for my pipeline.
+input_file = 'sample_data.mp4'
+
+# Choose a name for the output files
+base_file_name = 'output_file_17'
+
+# Settings for a low-res video derivative using a preset for a 320x240 resolution mp4 file
+low_res_video = { key: "#{base_file_name}.mp4", preset_id: '1351620000001-000061' }
+
+# Settings for a flash video derivative
+flash_video = { key: "#{base_file_name}.flv", preset_id: '1351620000001-100210' }
+
+# Settings to send to the Elastic Transcoder job
+job_settings = { pipeline_id: pipeline_id, output_key_prefix: "active_encode-demo_app/", outputs: [low_res_video, flash_video] }
+
+# Run the encoding
+Hydra::Derivatives::ActiveEncodeDerivatives.create(input_file, outputs: [job_settings])
+
+# Note: Your rails console will not return to the prompt until the encoding is complete,
+# so it might sit there for several minutes with no feedback.
+# Use the AWS console to see the current status of the encoding.
+```
+
+## How to create derivatives (One derivative per Elastic Transcoder job)
+
+If you want to run a separate Elastic Transcoder job for each derivative file, you could do something like this:
+
+```ruby
+# Settings for a low-res video derivative using a preset for a 320x240 resolution mp4 file.
+low_res_preset_id = '1351620000001-000061'
+low_res_output_file = 'output_15.mp4'
+low_res_video = { pipeline_id: pipeline_id, output_key_prefix: "active_encode-demo_app/", outputs: [{ key: low_res_output_file, preset_id: low_res_preset_id }] }
+
+# Settings for a flash video derivative
+flash_preset_id = '1351620000001-100210'
+flash_output_file = 'output_15.flv'
+flash_video = { pipeline_id: pipeline_id, output_key_prefix: "active_encode-demo_app/", outputs: [{ key: flash_output_file, preset_id: flash_preset_id }] }
+
+Hydra::Derivatives::ActiveEncodeDerivatives.create(input_file, outputs: [low_res_video, flash_video])
+```
+
+## How to pass in a ruby object
+
+If you want to pass in an `ActiveFedora::Base` object (or some other record) instead of just a String for the input file name, you need to set the `source` option to specify which method to call on your object to get the file name.  For example:
+
+```ruby
+# Some object that contains the source file name
+class Video
+  attr_accessor :source_file_name
+end
+
+video_record = Video.new
+video_record.source_file_name = 'sample_data.mp4'
+
+Hydra::Derivatives::ActiveEncodeDerivatives.create(video_record, source: :source_file_name, outputs: [low_res_video])
+```
+
+## How to pass in a custom encode class
+
+If you don't want to use the default encode class `::ActiveEncode::Base`, you can pass in `encode_class`:
+
+```ruby
+Hydra::Derivatives::ActiveEncodeDerivatives.create(video_record, encode_class: MyCustomEncode, source: :source_file_name, outputs: [low_res_video])
+```
+

data/hydra-derivatives.gemspec

@@ -23,10 +23,12 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "solr_wrapper", "~> 0.4"
   spec.add_development_dependency 'fcrepo_wrapper', '~> 0.2'
 
-  spec.add_dependency 'active-fedora', '>= 9.0', '< 12'
+  spec.add_dependency 'active-fedora', '>= 11.3.1', '< 12'
   spec.add_dependency 'mini_magick', '>= 3.2', '< 5'
   spec.add_dependency 'activesupport', '>= 4.0', '< 6'
   spec.add_dependency 'mime-types', '> 2.0', '< 4.0'
+  spec.add_dependency 'active_encode', '~>0.1'
+  spec.add_dependency 'addressable', '~>2.5'
   spec.add_dependency 'deprecation'
 end
 

data/lib/hydra/derivatives.rb

@@ -11,6 +11,7 @@ module Hydra
     # Runners take a single input and produce one or more outputs
     # The runner typically accomplishes this by using one or more processors
     autoload_under 'runners' do
+      autoload :ActiveEncodeDerivatives
       autoload :AudioDerivatives
       autoload :DocumentDerivatives
       autoload :FullTextExtract
@@ -30,8 +31,10 @@ module Hydra
 
     autoload_under 'services' do
       autoload :RetrieveSourceFileService
+      autoload :RemoteSourceFile
       autoload :PersistOutputFileService
       autoload :PersistBasicContainedOutputFileService
+      autoload :PersistExternalFileOutputFileService
       autoload :TempfileService
       autoload :MimeTypeService
     end
@@ -48,7 +51,7 @@ module Hydra
     end
 
     CONFIG_METHODS = [:ffmpeg_path, :libreoffice_path, :temp_file_base, :fits_path, :kdu_compress_path,
-                      :kdu_compress_recipes, :enable_ffmpeg, :source_file_service, :output_file_service].freeze
+                      :kdu_compress_recipes, :enable_ffmpeg, :source_file_service, :output_file_service, :active_encode_poll_time].freeze
     CONFIG_METHODS.each do |method|
       module_eval <<-RUBY
         def self.#{method}

data/lib/hydra/derivatives/config.rb

@@ -5,7 +5,8 @@ module Hydra
     class Config
       attr_writer :ffmpeg_path, :libreoffice_path, :temp_file_base,
                   :source_file_service, :output_file_service, :fits_path,
-                  :enable_ffmpeg, :kdu_compress_path, :kdu_compress_recipes
+                  :enable_ffmpeg, :kdu_compress_path, :kdu_compress_recipes,
+                  :active_encode_poll_time
 
       def ffmpeg_path
         @ffmpeg_path ||= 'ffmpeg'
@@ -72,6 +73,13 @@ module Hydra
             "Stiles={1024,1024}" ).gsub(/\s+/, " ").strip
         }
       end
+
+      # The poll time (in seconds) that the active encode
+      # processor will sleep before it checks the status of an
+      # encoding job.
+      def active_encode_poll_time
+        @active_encode_poll_time ||= 10
+      end
     end
   end
 end

data/lib/hydra/derivatives/processors.rb

@@ -6,6 +6,7 @@ module Hydra::Derivatives
       autoload :Processor
     end
 
+    autoload :ActiveEncode
     autoload :Audio
     autoload :Document
     autoload :Ffmpeg

data/lib/hydra/derivatives/processors/active_encode.rb

@@ -0,0 +1,56 @@
+require 'active_encode'
+
+module Hydra::Derivatives::Processors
+  class ActiveEncodeError < StandardError
+    def initialize(status, source_path, errors = [])
+      msg = "ActiveEncode status was \"#{status}\" for #{source_path}"
+      msg = "#{msg}: #{errors.join(' ; ')}" if errors.any?
+      super(msg)
+    end
+  end
+
+  class ActiveEncode < Processor
+    class_attribute :timeout
+    attr_accessor :encode_class
+    attr_reader :encode_job
+
+    def initialize(source_path, directives, opts = {})
+      super
+      @encode_class = opts.delete(:encode_class) || ::ActiveEncode::Base
+    end
+
+    def process
+      @encode_job = encode_class.create(source_path, directives)
+      timeout ? wait_for_encode_job_with_timeout : wait_for_encode_job
+      encode_job.output.each do |output|
+        output_file_service.call(output, directives)
+      end
+    end
+
+    private
+
+      def wait_for_encode_job_with_timeout
+        Timeout.timeout(timeout) { wait_for_encode_job }
+      rescue Timeout::Error
+        cleanup_after_timeout
+      end
+
+      # Wait until the encoding job is finished.  If the status
+      # is anything other than 'completed', raise an error.
+      def wait_for_encode_job
+        sleep Hydra::Derivatives.active_encode_poll_time while encode_job.reload.running?
+        raise ActiveEncodeError.new(encode_job.state, source_path, encode_job.errors) unless encode_job.completed?
+      end
+
+      # After a timeout error, try to cancel the encoding.
+      def cleanup_after_timeout
+        encode_job.cancel!
+      rescue => e
+        cancel_error = e
+      ensure
+        msg = "Unable to process ActiveEncode derivative: The command took longer than #{timeout} seconds to execute. Encoding will be cancelled."
+        msg = "#{msg} An error occurred while trying to cancel encoding: #{cancel_error}" if cancel_error
+        raise Hydra::Derivatives::TimeoutError, msg
+      end
+  end
+end

data/lib/hydra/derivatives/processors/document.rb

@@ -3,7 +3,7 @@ module Hydra::Derivatives::Processors
     include ShellBasedProcessor
 
     def self.encode(path, format, outdir)
-      execute "#{Hydra::Derivatives.libreoffice_path} --invisible --headless --convert-to #{format} --outdir #{outdir} #{path}"
+      execute "#{Hydra::Derivatives.libreoffice_path} --invisible --headless --convert-to #{format} --outdir #{outdir} #{Shellwords.escape(path)}"
     end
 
     # Converts the document to the format specified in the directives hash.

data/lib/hydra/derivatives/processors/ffmpeg.rb

@@ -14,7 +14,7 @@ module Hydra::Derivatives::Processors
       def encode(path, options, output_file)
         inopts = options[INPUT_OPTIONS] ||= "-y"
         outopts = options[OUTPUT_OPTIONS] ||= ""
-        execute "#{Hydra::Derivatives.ffmpeg_path} #{inopts} -i \"#{path}\" #{outopts} #{output_file}"
+        execute "#{Hydra::Derivatives.ffmpeg_path} #{inopts} -i #{Shellwords.escape(path)} #{outopts} #{output_file}"
       end
     end
   end

data/lib/hydra/derivatives/processors/jpeg2k_image.rb

@@ -75,7 +75,7 @@ module Hydra::Derivatives::Processors
 
       def encode(path, recipe, output_file)
         kdu_compress = Hydra::Derivatives.kdu_compress_path
-        execute "#{kdu_compress} -quiet -i #{path} -o #{output_file} #{recipe}"
+        execute "#{kdu_compress} -quiet -i #{Shellwords.escape(path)} -o #{output_file} #{recipe}"
       end
 
       def tmp_file(ext)

data/lib/hydra/derivatives/runners/active_encode_derivatives.rb

@@ -0,0 +1,44 @@
+module Hydra::Derivatives
+  class ActiveEncodeDerivatives < Runner
+    # @param [String, ActiveFedora::Base] object_or_filename source file name (or path), or an object that has a method that will return the file name
+    # @param [Hash] options options to pass to the encoder
+    # @option options [Symbol] :source a method that can be called on the object to retrieve the source file's name
+    # @option options [Symbol] :encode_class class name of the encode object (usually a subclass of ::ActiveEncode::Base)
+    # @options options [Array] :outputs a list of desired outputs
+    def self.create(object_or_filename, options)
+      processor_opts = processor_options(options)
+      source_file(object_or_filename, options) do |file_name|
+        transform_directives(options.delete(:outputs)).each do |instructions|
+          processor = processor_class.new(file_name, instructions, processor_opts)
+          processor.process
+        end
+      end
+    end
+
+    # Use the source service configured for this class or default to the remote file service
+    def self.source_file_service
+      @source_file_service || RemoteSourceFile
+    end
+
+    # Use the output service configured for this class or default to the external file service
+    def self.output_file_service
+      @output_file_service || PersistExternalFileOutputFileService
+    end
+
+    def self.processor_class
+      Processors::ActiveEncode
+    end
+
+    class << self
+      private
+
+        # Build an options hash specifically for the processor isolated from the runner options
+        def processor_options(options)
+          opts = { output_file_service: output_file_service }
+          encode_class = options.delete(:encode_class)
+          opts = opts.merge(encode_class: encode_class) if encode_class
+          opts
+        end
+    end
+  end
+end

data/lib/hydra/derivatives/services/persist_basic_contained_output_file_service.rb

@@ -9,31 +9,55 @@ module Hydra::Derivatives
     #
     # NOTE: Uses basic containment. If you want to use direct containment (ie. with PCDM) you must use a different service (ie. Hydra::Works::AddFileToGenericFile Service)
     #
-    # @param [#read] stream the data to be persisted
+    # @param [IO,String] content the data to be persisted
     # @param [Hash] directives directions which can be used to determine where to persist to.
     # @option directives [String] url This can determine the path of the object.
-    def self.call(stream, directives)
-      file = Hydra::Derivatives::IoDecorator.new(stream, new_mime_type(directives.fetch(:format)))
-      o_name = determine_original_name(file)
-      m_type = determine_mime_type(file)
-      uri = URI(directives.fetch(:url))
-      raise ArgumentError, "#{uri} is not an http uri" unless uri.scheme == 'http'
-      remote_file = ActiveFedora::File.new(uri.to_s)
+    # @option directives [String] format The file extension (e.g. 'jpg')
+    def self.call(content, directives)
+      file = io(content, directives)
+      remote_file = retrieve_remote_file(directives)
       remote_file.content = file
-      remote_file.mime_type = m_type
-      remote_file.original_name = o_name
+      remote_file.mime_type = determine_mime_type(file)
+      remote_file.original_name = determine_original_name(file)
       remote_file.save
     end
 
-    def self.new_mime_type(format)
-      case format
+    # Override this implementation if you need a remote file from a different location
+    # @return [ActiveFedora::File]
+    def self.retrieve_remote_file(directives)
+      uri = URI(directives.fetch(:url))
+      raise ArgumentError, "#{uri} is not an http uri" unless uri.scheme == 'http'
+      ActiveFedora::File.new(uri.to_s)
+    end
+    private_class_method :retrieve_remote_file
+
+    def self.io(content, directives)
+      Hydra::Derivatives::IoDecorator.new(content, new_mime_type(directives.fetch(:format), charset(content)))
+    end
+    private_class_method :io
+
+    def self.new_mime_type(extension, charset = nil)
+      fmt = mime_format(extension)
+      fmt += "; charset=#{charset}" if charset
+      fmt
+    end
+
+    # Strings (from FullText) have encoding. Retrieve it
+    def self.charset(content)
+      content.encoding.name if content.respond_to?(:encoding)
+    end
+    private_class_method :charset
+
+    def self.mime_format(extension)
+      case extension
       when 'mp4'
         'video/mp4' # default is application/mp4
       when 'webm'
         'video/webm' # default is audio/webm
       else
-        MIME::Types.type_for(format).first.to_s
+        MIME::Types.type_for(extension).first.to_s
       end
     end
+    private_class_method :mime_format
   end
 end

data/lib/hydra/derivatives/services/persist_external_file_output_file_service.rb

@@ -0,0 +1,20 @@
+require 'addressable'
+
+module Hydra::Derivatives
+  class PersistExternalFileOutputFileService < PersistOutputFileService
+    # Persists a new file at specified location that points to external content
+    # @param [Hash] output information about the external derivative file
+    # @option output [String] url the location of the external content
+    # @param [Hash] directives directions which can be used to determine where to persist to.
+    # @option directives [String] url This can determine the path of the object.
+    def self.call(output, directives)
+      external_file = ActiveFedora::File.new(directives[:url])
+      # TODO: Replace the following two lines with the shorter call to #external_url once active_fedora/pull/1234 is merged
+      external_file.content = ''
+      external_file.mime_type = "message/external-body; access-type=URL; URL=\"#{output[:url]}\""
+      # external_file.external_url = output[:url]
+      external_file.original_name = Addressable::URI.parse(output[:url]).path.split('/').last
+      external_file.save
+    end
+  end
+end

data/lib/hydra/derivatives/services/remote_source_file.rb

@@ -0,0 +1,18 @@
+# For the case where the source file is a remote file, and we
+# don't want to download the file locally, just return the
+# file name or file path (or whatever we need to pass to the
+# encoding service so that it can find the file).
+
+module Hydra::Derivatives
+  class RemoteSourceFile
+    # Finds the file name of the remote source file.
+    # @param [String, ActiveFedora::Base] object file name, or an object that has a method that will return the file name
+    # @param [Hash] options
+    # @option options [Symbol] :source a method that can be called on the object to retrieve the source file's name
+    # @yield [String] the file name
+    def self.call(object, options, &_block)
+      source_name = options.fetch(:source, :to_s)
+      yield(object.send(source_name))
+    end
+  end
+end

data/spec/processors/active_encode_spec.rb

@@ -0,0 +1,132 @@
+require 'spec_helper'
+
+describe Hydra::Derivatives::Processors::ActiveEncode do
+  let(:file_path) { File.join(fixture_path, 'videoshort.mp4') }
+  let(:directives) { { url: '12345/derivative' } }
+  let(:output_file_service) { Hydra::Derivatives::PersistExternalFileOutputFileService }
+  let(:options) { { output_file_service: output_file_service } }
+  let(:processor) { described_class.new(file_path, directives, options) }
+
+  describe '#process' do
+    subject { processor.process }
+
+    # Mock out the actual encoding, just pretend that the
+    # encode finished and returned a certain status.
+    let(:failed_status) { false }
+    let(:cancelled_status) { false }
+    let(:completed_status) { false }
+    let(:state) { :completed }
+    let(:errors) { [] }
+    let(:external_url) { 'http://www.example.com/external/content' }
+    let(:output) { [{ url: external_url }] }
+    let(:encode_job_double) do
+      enc = double('encode_job',
+                   state: state,
+                   errors: errors,
+                   output: output,
+                   running?: false,
+                   completed?: completed_status,
+                   failed?: failed_status,
+                   cancelled?: cancelled_status)
+      allow(enc).to receive(:reload).and_return(enc)
+      enc
+    end
+
+    context 'with a custom encode class' do
+      before do
+        class TestEncode < ::ActiveEncode::Base; end
+
+        # For this spec we don't care what happens with output,
+        # so stub it out to speed up the spec.
+        allow(output_file_service).to receive(:call)
+      end
+
+      after { Object.send(:remove_const, :TestEncode) }
+
+      let(:completed_status) { true }
+      let(:state) { :completed }
+      let(:options) do
+        {
+          encode_class: TestEncode,
+          output_file_service: output_file_service
+        }
+      end
+
+      it 'uses the configured encode class' do
+        expect(TestEncode).to receive(:create).and_return(encode_job_double)
+        subject
+      end
+    end
+
+    context 'when the encoding failed' do
+      let(:state) { :failed }
+      let(:failed_status) { true }
+      let(:errors) { ['error 1', 'error 2'] }
+
+      before do
+        # Don't really encode the file during specs
+        allow(::ActiveEncode::Base).to receive(:create).and_return(encode_job_double)
+      end
+
+      it 'raises an exception' do
+        expect { subject }.to raise_error(Hydra::Derivatives::Processors::ActiveEncodeError, "ActiveEncode status was \"failed\" for #{file_path}: error 1 ; error 2")
+      end
+    end
+
+    context 'when the encoding was cancelled' do
+      let(:state) { :cancelled }
+      let(:cancelled_status) { true }
+
+      before do
+        # Don't really encode the file during specs
+        allow(::ActiveEncode::Base).to receive(:create).and_return(encode_job_double)
+      end
+
+      it 'raises an exception' do
+        expect { subject }.to raise_error(Hydra::Derivatives::Processors::ActiveEncodeError, "ActiveEncode status was \"cancelled\" for #{file_path}")
+      end
+    end
+
+    context 'when the timeout is set' do
+      before do
+        processor.timeout = 0.01
+        allow(processor).to receive(:wait_for_encode_job) { sleep 0.1 }
+      end
+
+      it 'raises a timeout exception' do
+        msg = "Unable to process ActiveEncode derivative: The command took longer than 0.01 seconds to execute. Encoding will be cancelled."
+        expect { processor.process }.to raise_error Hydra::Derivatives::TimeoutError, msg
+      end
+    end
+
+    context 'when the timeout is not set' do
+      before do
+        processor.timeout = nil
+        # Don't really encode the file during specs
+        allow(::ActiveEncode::Base).to receive(:create).and_return(encode_job_double)
+      end
+
+      it 'processes the encoding without a timeout' do
+        expect(processor).not_to receive(:wait_for_encode_job_with_timeout)
+        expect(processor).to receive(:wait_for_encode_job).once
+        processor.process
+      end
+    end
+
+    context 'when error occurs during timeout cleanup' do
+      let(:error) { StandardError.new('some error message') }
+
+      before do
+        processor.timeout = 0.01
+        allow(processor).to receive(:wait_for_encode_job) { sleep 0.1 }
+        allow(::ActiveEncode::Base).to receive(:create).and_return(encode_job_double)
+        allow(encode_job_double).to receive(:cancel!).and_raise(error)
+      end
+
+      it 'doesnt lose the timeout error, but adds the new error message' do
+        msg = "Unable to process ActiveEncode derivative: The command took longer than 0.01 seconds to execute. Encoding will be cancelled. An error occurred while trying to cancel encoding: some error message"
+        expect { processor.process }.to raise_error Hydra::Derivatives::TimeoutError, msg
+      end
+    end
+  end
+end

data/spec/processors/full_text_spec.rb

@@ -7,7 +7,10 @@ describe Hydra::Derivatives::Processors::FullText do
 
   describe "#process" do
     it 'extracts fulltext and stores the results' do
-      expect(processor.output_file_service).to receive(:call).with(/Project Charter for E-Content Delivery Platform Review/, directives)
+      expect(processor.output_file_service).to receive(:call) do |first, second|
+        expect(first).to match(/Project Charter for E-Content Delivery Platform Review/)
+        expect(second).to eq directives
+      end
       processor.process
     end
   end

data/spec/runners/active_encode_derivatives_spec.rb

@@ -0,0 +1,38 @@
+require 'spec_helper'
+
+describe Hydra::Derivatives::ActiveEncodeDerivatives do
+  context '.create' do
+    before do
+      class TestVideo < ActiveFedora::Base
+        attr_accessor :remote_file_name
+      end
+    end
+
+    after { Object.send(:remove_const, :TestVideo) }
+
+    let(:file_path) { 'some/path/to/my_video.mp4' }
+    let(:video_record) { TestVideo.new(remote_file_name: file_path) }
+    let(:options) { { source: :remote_file_name, outputs: [low_res_video] } }
+    let(:low_res_video) { { some_key: 'some options to pass to my encoder service' } }
+    let(:processor) { double('processor') }
+
+    it 'calls the processor with the right arguments' do
+      expect(Hydra::Derivatives::Processors::ActiveEncode).to receive(:new).with(file_path, low_res_video, output_file_service: Hydra::Derivatives::PersistExternalFileOutputFileService).and_return(processor)
+      expect(processor).to receive(:process)
+      described_class.create(video_record, options)
+    end
+
+    context 'with a custom encode class' do
+      before { class TestEncode < ::ActiveEncode::Base; end }
+      after { Object.send(:remove_const, :TestEncode) }
+
+      let(:options) { { encode_class: TestEncode, source: :remote_file_name, outputs: [low_res_video] } }
+
+      it 'calls the processor with the right arguments' do
+        expect(Hydra::Derivatives::Processors::ActiveEncode).to receive(:new).with(file_path, low_res_video, output_file_service: Hydra::Derivatives::PersistExternalFileOutputFileService, encode_class: TestEncode).and_return(processor)
+        expect(processor).to receive(:process)
+        described_class.create(video_record, options)
+      end
+    end
+  end
+end

data/spec/services/persist_basic_contained_output_file_service_spec.rb

@@ -17,11 +17,24 @@ describe Hydra::Derivatives::PersistBasicContainedOutputFileService do
   # not be sufficient.
   context "when file is basic contained (default assumption)" do
     let(:object) { BasicContainerObject.create }
-    let(:stream) { StringIO.new("fake file content") }
-    it "persists the file to the specified destination on the given object" do
-      described_class.call(stream, format: 'jpg', url: "#{object.uri}/the_derivative_name")
-      expect(object.send(destination_name.to_sym).content).to eq("fake file content")
-      expect(object.send(destination_name.to_sym).content_changed?).to eq false
+    let(:content) { StringIO.new("fake file content") }
+    let(:resource) { object.public_send(destination_name.to_sym) }
+    context "and the content is a stream" do
+      it "persists the file to the specified destination on the given object" do
+        described_class.call(content, format: 'jpg', url: "#{object.uri}/the_derivative_name")
+        expect(resource.content).to start_with("fake file content")
+        expect(resource.content_changed?).to eq false
+        expect(resource.mime_type).to eq 'image/jpeg'
+      end
+    end
+
+    context "and content is a string" do
+      let(:content) { "fake file content - ÅÄÖ" }
+      it "persists the file to the specified destination on the given object" do
+        described_class.call(content, format: 'txt', url: "#{object.uri}/the_derivative_name")
+        expect(resource.content).to eq("fake file content - ÅÄÖ")
+        expect(resource.mime_type).to eq 'text/plain;charset=UTF-8'
+      end
     end
   end
 end

data/spec/services/persist_external_file_output_file_service_spec.rb

@@ -0,0 +1,26 @@
+require 'spec_helper'
+
+describe Hydra::Derivatives::PersistExternalFileOutputFileService do
+  before do
+    class ExternalDerivativeContainerObject < ActiveFedora::Base
+      has_subresource "external_derivative"
+    end
+  end
+  after do
+    Object.send(:remove_const, :ExternalDerivativeContainerObject)
+  end
+
+  let(:object)            { ExternalDerivativeContainerObject.create }
+  let(:directives)        { { url: "#{object.uri}/external_derivative" } }
+  let(:external_url)      { 'http://www.example.com/external/content' }
+  let(:output)            { { url: external_url } }
+  let(:destination_name)  { 'external_derivative' }
+
+  describe '.call' do
+    it "persists the external file to the specified destination on the given object" do
+      described_class.call(output, directives)
+      expect(object.send(destination_name.to_sym).mime_type).to eq "message/external-body;access-type=URL;url=\"http://www.example.com/external/content\""
+      expect(object.send(destination_name.to_sym).content).to eq ''
+    end
+  end
+end

data/spec/services/remote_source_file_spec.rb

@@ -0,0 +1,33 @@
+require 'spec_helper'
+
+class TestObject < ActiveFedora::Base
+  attr_accessor :source_file_name
+end
+
+describe Hydra::Derivatives::RemoteSourceFile do
+  describe '.call' do
+    let(:file_name) { 'my_source_file.mp4' }
+
+    context 'when you pass in a String file name' do
+      let(:input_obj) { file_name }
+      let(:options) { Hash.new }
+
+      it 'it yields the file name' do
+        expect do |blk|
+          described_class.call(input_obj, options, &blk)
+        end.to yield_with_args(file_name)
+      end
+    end
+
+    context 'when you pass in an ActiveFedora::Base object ' do
+      let(:input_obj) { TestObject.new(source_file_name: file_name) }
+      let(:options) { { source: :source_file_name } }
+
+      it 'it yields the file name' do
+        expect do |blk|
+          described_class.call(input_obj, options, &blk)
+        end.to yield_with_args(file_name)
+      end
+    end
+  end
+end

data/spec/units/config_spec.rb

@@ -41,4 +41,10 @@ describe "the configuration" do
     subject.source_file_service = source_file_service
     expect(subject.source_file_service).to eq(source_file_service)
   end
+
+  it "lets you set the poll time for ActiveEncode jobs" do
+    expect(subject.active_encode_poll_time).to eq 10
+    subject.active_encode_poll_time = 15
+    expect(subject.active_encode_poll_time).to eq 15
+  end
 end

data/spec/units/derivatives_spec.rb

@@ -53,6 +53,10 @@ describe Hydra::Derivatives do
       subject.kdu_compress_path = '/usr/local/bin/kdu_compress'
       subject.reset_config!
       expect(subject.kdu_compress_path).to eq('kdu_compress')
+
+      subject.active_encode_poll_time = 2
+      subject.reset_config!
+      expect(subject.active_encode_poll_time).to eq 10
     end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hydra-derivatives
 version: !ruby/object:Gem::Version
-  version: 3.2.2
+  version: 3.3.0
 platform: ruby
 authors:
 - Justin Coyne
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-04-07 00:00:00.000000000 Z
+date: 2017-06-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -86,7 +86,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '9.0'
+        version: 11.3.1
     - - "<"
       - !ruby/object:Gem::Version
         version: '12'
@@ -96,7 +96,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '9.0'
+        version: 11.3.1
     - - "<"
       - !ruby/object:Gem::Version
         version: '12'
@@ -160,6 +160,34 @@ dependencies:
     - - "<"
       - !ruby/object:Gem::Version
         version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: active_encode
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: addressable
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.5'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.5'
 - !ruby/object:Gem::Dependency
   name: deprecation
   requirement: !ruby/object:Gem::Requirement
@@ -195,6 +223,7 @@ files:
 - TODO
 - VERSION
 - config/jetty.yml
+- doc/amazon_elastic_transcoder.md
 - hydra-derivatives.gemspec
 - lib/color_profiles/license.txt
 - lib/color_profiles/sRGB_IEC61966-2-1_no_black_scaling.icc
@@ -204,6 +233,7 @@ files:
 - lib/hydra/derivatives/io_decorator.rb
 - lib/hydra/derivatives/logger.rb
 - lib/hydra/derivatives/processors.rb
+- lib/hydra/derivatives/processors/active_encode.rb
 - lib/hydra/derivatives/processors/audio.rb
 - lib/hydra/derivatives/processors/document.rb
 - lib/hydra/derivatives/processors/ffmpeg.rb
@@ -216,6 +246,7 @@ files:
 - lib/hydra/derivatives/processors/video.rb
 - lib/hydra/derivatives/processors/video/config.rb
 - lib/hydra/derivatives/processors/video/processor.rb
+- lib/hydra/derivatives/runners/active_encode_derivatives.rb
 - lib/hydra/derivatives/runners/audio_derivatives.rb
 - lib/hydra/derivatives/runners/document_derivatives.rb
 - lib/hydra/derivatives/runners/full_text_extract.rb
@@ -227,7 +258,9 @@ files:
 - lib/hydra/derivatives/services/capability_service.rb
 - lib/hydra/derivatives/services/mime_type_service.rb
 - lib/hydra/derivatives/services/persist_basic_contained_output_file_service.rb
+- lib/hydra/derivatives/services/persist_external_file_output_file_service.rb
 - lib/hydra/derivatives/services/persist_output_file_service.rb
+- lib/hydra/derivatives/services/remote_source_file.rb
 - lib/hydra/derivatives/services/retrieve_source_file_service.rb
 - lib/hydra/derivatives/services/tempfile_service.rb
 - solr/config/_rest_managed.json
@@ -261,6 +294,7 @@ files:
 - spec/fixtures/test.xls
 - spec/fixtures/test.xlsx
 - spec/fixtures/world.png
+- spec/processors/active_encode_spec.rb
 - spec/processors/document_spec.rb
 - spec/processors/full_text_spec.rb
 - spec/processors/image_spec.rb
@@ -268,8 +302,11 @@ files:
 - spec/processors/processor_spec.rb
 - spec/processors/shell_based_processor_spec.rb
 - spec/processors/video_spec.rb
+- spec/runners/active_encode_derivatives_spec.rb
 - spec/services/audio_derivatives_spec.rb
 - spec/services/persist_basic_contained_output_file_service_spec.rb
+- spec/services/persist_external_file_output_file_service_spec.rb
+- spec/services/remote_source_file_spec.rb
 - spec/services/retrieve_source_file_service_spec.rb
 - spec/services/tempfile_service_spec.rb
 - spec/spec_helper.rb
@@ -299,7 +336,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.10
+rubygems_version: 2.6.12
 signing_key: 
 specification_version: 4
 summary: Derivative generation plugin for hydra
@@ -319,6 +356,7 @@ test_files:
 - spec/fixtures/test.xls
 - spec/fixtures/test.xlsx
 - spec/fixtures/world.png
+- spec/processors/active_encode_spec.rb
 - spec/processors/document_spec.rb
 - spec/processors/full_text_spec.rb
 - spec/processors/image_spec.rb
@@ -326,8 +364,11 @@ test_files:
 - spec/processors/processor_spec.rb
 - spec/processors/shell_based_processor_spec.rb
 - spec/processors/video_spec.rb
+- spec/runners/active_encode_derivatives_spec.rb
 - spec/services/audio_derivatives_spec.rb
 - spec/services/persist_basic_contained_output_file_service_spec.rb
+- spec/services/persist_external_file_output_file_service_spec.rb
+- spec/services/remote_source_file_spec.rb
 - spec/services/retrieve_source_file_service_spec.rb
 - spec/services/tempfile_service_spec.rb
 - spec/spec_helper.rb