active_encode 0.4.1 → 0.8.0

data/README.md

@@ -1,6 +1,21 @@
 # ActiveEncode
 
-This gem serves as the basis for the interface between a Ruby (Rails) application and a provider of transcoding services such as [Opencast Matterhorn](http://opencast.org), [Zencoder](http://zencoder.com), and [Amazon Elastic Transcoder](http://aws.amazon.com/elastictranscoder/).
+Code: [![Version](https://badge.fury.io/rb/active_encode.png)](http://badge.fury.io/rb/active_encode)
+[![Build Status](https://travis-ci.org/samvera-labs/active_encode.png?branch=master)](https://travis-ci.org/samvera-labs/active_encode)
+[![Coverage Status](https://coveralls.io/repos/github/samvera-labs/active_encode/badge.svg?branch=master)](https://coveralls.io/github/samvera-labs/active_encode?branch=master)
+
+Docs: [![Contribution Guidelines](http://img.shields.io/badge/CONTRIBUTING-Guidelines-blue.svg)](./CONTRIBUTING.md)
+[![Apache 2.0 License](http://img.shields.io/badge/APACHE2-license-blue.svg)](./LICENSE)
+
+Jump in: [![Slack Status](http://slack.samvera.org/badge.svg)](http://slack.samvera.org/)
+
+# What is ActiveEncode?
+
+ActiveEncode serves as the basis for the interface between a Ruby (Rails) application and a provider of encoding services such as [FFmpeg](https://www.ffmpeg.org/), [Amazon Elastic Transcoder](http://aws.amazon.com/elastictranscoder/), and [Zencoder](http://zencoder.com).
+
+# Help
+
+The Samvera community is here to help. Please see our [support guide](./SUPPORT.md).
 
 ## Installation
 
@@ -18,41 +33,76 @@ Or install it yourself as:
 
     $ gem install active_encode
 
+## Prerequisites
+
+FFmpeg (tested with version 4+) and mediainfo (version 17.10+) need to be installed to use the FFmpeg engine adapter.
+
 ## Usage
 
 Set the engine adapter (default: test), configure it (if neccessary), then submit encoding jobs!
 
 ```ruby
-ActiveEncode::Base.engine_adapter = :matterhorn
-ActiveEncode::Base.create(File.open('spec/fixtures/Bars_512kb.mp4'))
+ActiveEncode::Base.engine_adapter = :ffmpeg
+file = "file://#{File.absolute_path "spec/fixtures/fireworks.mp4"}"
+ActiveEncode::Base.create(file, { outputs: [{ label: "low", ffmpeg_opt: "-s 640x480", extension: "mp4"}, { label: "high", ffmpeg_opt: "-s 1280x720", extension: "mp4"}] })
 ```
-Create returns an encoding job that has been submitted to the encoding engine for processing.  At this point it will have an id, a state, the input, and any additional information the encoding engine returns.
+Create returns an encoding job that has been submitted to the adapter for processing.  At this point it will have an id, a state, the input, and any additional information the adapter returns.
 
 ```ruby
-#<ActiveEncode::Base:0x00000003f3cd90 @input="http://localhost:8080/files/mediapackage/edcac316-1f98-44b1-88ca-0ce6f80aebc0/ff43c56f-7b8f-4d9c-a846-6e51de2e8cb4/Bars_512kb.mp4", @options={:preset=>"avalon", :stream_base=>"file:///home/cjcolvar/Code/avalon/avalon/red5/webapps/avalon/streams"}, @id="12154", @state=:running, @current_operations=[], @percent_complete=0.0, @output=[], @errors=[], @tech_metadata={}>
+#<ActiveEncode::Base:0x007f8ef3b2ae88 @input=#<ActiveEncode::Input:0x007f8ef3b23188 @url="file:///Users/cjcolvar/Documents/Code/samvera-labs/active_encode/spec/fixtures/fireworks.mp4", @width=960.0, @height=540.0, @frame_rate=29.671, @duration=6024, @file_size=1629578, @audio_codec="mp4a-40-2", @video_codec="avc1", @audio_bitrate=69737, @video_bitrate=2092780, @created_at=2018-12-03 14:22:05 -0500, @updated_at=2018-12-03 14:22:05 -0500, @id=7653>, @options={:outputs=>[{:label=>"low", :ffmpeg_opt=>"-s 640x480", :extension=>"mp4"}, {:label=>"high", :ffmpeg_opt=>"-s 1280x720", :extension=>"mp4"}]}, @id="1e4a907a-ccff-494f-ad70-b1c5072c2465", @created_at=2018-12-03 14:22:05 -0500, @updated_at=2018-12-03 14:22:05 -0500, @current_operations=[], @output=[], @state=:running, @percent_complete=1, @errors=[]>
 ```
 ```ruby
-encode.id  # "12103"
+encode.id  # "1e4a907a-ccff-494f-ad70-b1c5072c2465"
 encode.state  # :running
 ```
 
-This encode can be looked back up later using #find.  Alternatively, use #reload to refresh an instance with the latest information from the
+This encode can be looked back up later using #find.  Alternatively, use #reload to refresh an instance with the latest information from the adapter:
 
 ```ruby
-encode = ActiveEncode::Base.find("12103")
+encode = ActiveEncode::Base.find("1e4a907a-ccff-494f-ad70-b1c5072c2465")
 encode.reload
 ```
 
-Progress of a running encode is shown with a current operation (multiple possible if outputs are generated in parallel) and percent complete.  Technical metadata about the input file is added by the encoding engine.  This should include a mime type, checksum, duration, and basic technical details of the audio and video content of the file (codec, audio channels, bitrate, framerate, and dimensions).  Outputs are added once they are created and should include the same technical metadata along with an id, label, and url.
+Progress of a running encode is shown with current operations (multiple are possible when outputs are generated in parallel) and percent complete.  Technical metadata about the input file may be added by the adapter.  This should include a mime type, checksum, duration, and basic technical details of the audio and video content of the file (codec, audio channels, bitrate, frame rate, and dimensions).  Outputs are added once they are created and should include the same technical metadata along with an id, label, and url.
 
-If the encoding job should be stopped, call cancel:
+If you want to stop the encoding job call cancel:
 
 ```ruby
 encode.cancel!
 encode.cancelled?  # true
 ```
 
-An encoding job is meant to be the record of the work of the encoding engine and not the current state of the outputs.  Therefore removed outputs will not be reflected in the encoding job.
+An encoding job is meant to be the record of the work of the encoding engine and not the current state of the outputs.  Therefore moved or deleted outputs will not be reflected in the encoding job.
+
+### AWS ElasticTranscoder
+
+To use active_encode with the AWS ElasticTransoder, the following are required:
+- An S3 bucket to store master files
+- An S3 bucket to store derivatives (recommended to be separate)
+- An ElasticTranscoder pipeline
+- Some transcoding presets for the pipeline
+
+Set the adapter:
+
+```ruby
+ActiveEncode::Base.engine_adapter = :elastic_transcoder
+```
+
+Construct the options hash:
+
+```ruby
+outputs = [{ key: "quality-low/hls/fireworks", preset_id: '1494429796844-aza6zh', segment_duration: '2' },
+           { key: "quality-medium/hls/fireworks", preset_id: '1494429797061-kvg9ki', segment_duration: '2' },
+           { key: "quality-high/hls/fireworks", preset_id: '1494429797265-9xi831', segment_duration: '2' }]
+options = {pipeline_id: 'my-pipeline-id', masterfile_bucket: 'my-master-files', outputs: outputs}
+```
+
+Create the job:
+
+```ruby
+file = 'file:///path/to/file/fireworks.mp4' # or 's3://my-bucket/fireworks.mp4'
+encode = ActiveEncode::Base.create(file, options)
+```
 
 ### Custom jobs
 
@@ -72,19 +122,80 @@ end
 
 ### Engine Adapters
 
-Engine adapters are shims between ActiveEncode and the back end encoding service.  Each service has its own API and idiosyncracies so consult the table below to see what features are supported by each adapter.  Add an additional engines by creating an engine adapter class that implements :create, :find, and :cancel.
+Engine adapters are shims between ActiveEncode and the back end encoding service.  You can add an additional engine by creating an engine adapter class that implements `:create`, `:find`, and `:cancel` and passes the shared specs.
+
+For example:
+```ruby
+# In your application at:
+# lib/active_encode/engine_adapters/my_custom_adapter.rb
+module ActiveEncode
+  module EngineAdapters
+    class MyCustomAdapter
+      def create(input_url, options = {})
+        # Start a new encoding job. This may be an external service, or a
+        # locally queued job.
+
+        # Return an instance ActiveEncode::Base (or subclass) that represents
+        # the encoding job that was just started.        
+      end
+
+      def find(id, opts = {})
+        # Find the encoding job for the given parameters.
+
+        # Return an instance of ActiveEncode::Base (or subclass) that represents
+        # the found encoding job.
+      end
+
+      def cancel(id)
+        # Cancel the encoding job for the given id.
+
+        # Return an instance of ActiveEncode::Base (or subclass) that represents
+        # the canceled job.
+      end
+    end
+  end
+end
+```
+Then, use the shared specs...
+```ruby
+# In your application at...
+# spec/lib/active_encode/engine_adapters/my_custom_adapter_spec.rb
+require 'spec_helper'
+require 'active_encode/spec/shared_specs'
+RSpec.describe MyCustomAdapter do
+  let(:created_job) {
+    # an instance of ActiveEncode::Base represented a newly created encode job
+  }
+  let(:running_job) {
+    # an instance of ActiveEncode::Base represented a running encode job
+  }
+  let(:canceled_job) {
+    # an instance of ActiveEncode::Base represented a canceled encode job
+  }
+  let(:completed_job) {
+    # an instance of ActiveEncode::Base represented a completed encode job
+  }
+  let(:failed_job) {
+    # an instance of ActiveEncode::Base represented a failed encode job
+  }
+  let(:completed_tech_metadata) {
+    # a hash representing completed technical metadata
+  }
+  let(:completed_output) {
+    # data representing completed output
+  }
+  let(:failed_tech_metadata) {
+    # a hash representing failed technical metadata
+  }
+
+  # Run the shared specs.
+  it_behaves_like 'an ActiveEncode::EngineAdapter'
+end
+```
 
-| Adapter/Feature          | Create | Find | Cancel | Preset | Multiple Outputs |
-|--------------------------|--------|------|--------|--------|------------------|
-| Test                     |    X   |  X   |   X    |        |                  |
-| AWS Elastic Transcoder   |    X   |  X   |   X    |        |                  |
-| Zencoder                 |    X   |  X   |   X    |        |                  |
-| Matterhorn               |    X   |  X   |   X    |   X    |         X        |
+# Acknowledgments
 
-## Contributing
+This software has been developed by and is brought to you by the Samvera community.  Learn more at the
+[Samvera website](http://samvera.org/).
 
-1. Fork it ( https://github.com/projecthydra-labs/active_encode/fork )
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create a new Pull Request
+![Samvera Logo](https://wiki.duraspace.org/download/thumbnails/87459292/samvera-fall-font2-200w.png?version=1&modificationDate=1498550535816&api=v2)

data/SUPPORT.md

@@ -0,0 +1,5 @@
+If you would like to report an issue, first search [the list of issues](https://github.com/samvera-labs/active_encode/issues/) to see if someone else has already reported it, and then feel free to [create a new issue](https://github.com/samvera-labs/active_encode/issues/new).
+
+If you have questions or need help, please email [the Samvera community tech list](https://groups.google.com/forum/#!forum/samvera-tech) or stop by the #dev channel in [the Samvera community Slack team](https://wiki.duraspace.org/pages/viewpage.action?pageId=87460391#Getintouch!-Slack).
+
+You can learn more about the various Samvera communication channels on the [Get in touch!](https://wiki.duraspace.org/pages/viewpage.action?pageId=87460391) wiki page.

data/active_encode.gemspec

@@ -1,4 +1,5 @@
 # coding: utf-8
+# frozen_string_literal: true
 
 lib = File.expand_path('../lib', __FILE__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
@@ -10,7 +11,7 @@ Gem::Specification.new do |spec|
   spec.authors       = ["Michael Klein, Chris Colvard, Phuong Dinh"]
   spec.email         = ["mbklein@gmail.com, chris.colvard@gmail.com, phuongdh@gmail.com"]
   spec.summary       = 'Declare encode job classes that can be run by a variety of encoding services'
-  spec.description   = 'This gem serves as the basis for the interface between a Ruby (Rails) application and a provider of transcoding services such as Opencast Matterhorn, Zencoder, and Amazon Elastic Transcoder.'
+  spec.description   = 'This gem provides an interface to transcoding services such as Ffmpeg, Amazon Elastic Transcoder, or Zencoder.'
   spec.homepage      = "https://github.com/samvera-labs/active_encode"
   spec.license       = "Apache-2.0"
 
@@ -21,12 +22,21 @@ Gem::Specification.new do |spec|
 
   spec.add_dependency "rails"
 
+  spec.add_development_dependency "aws-sdk-cloudwatchevents"
+  spec.add_development_dependency "aws-sdk-cloudwatchlogs"
+  spec.add_development_dependency "aws-sdk-elastictranscoder"
+  spec.add_development_dependency "aws-sdk-mediaconvert"
+  spec.add_development_dependency "aws-sdk-s3"
+  spec.add_development_dependency "bixby", '~> 1.0.0'
   spec.add_development_dependency "bundler"
   spec.add_development_dependency "coveralls"
   spec.add_development_dependency "database_cleaner"
-  spec.add_development_dependency "engine_cart"
+  spec.add_development_dependency "engine_cart", "~> 2.2"
   spec.add_development_dependency "rake"
   spec.add_development_dependency "rspec"
-  spec.add_development_dependency "rspec-its"
+  spec.add_development_dependency 'rspec_junit_formatter'
   spec.add_development_dependency "rspec-rails"
+
+  # Pin sprockets to < 4 so it works with ruby 2.5+
+  spec.add_dependency 'sprockets', '< 4'
 end

data/app/controllers/active_encode/encode_record_controller.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+module ActiveEncode
+  class EncodeRecordController < ActionController::Base
+    rescue_from ActiveRecord::RecordNotFound do |e|
+      render json: { message: e.message }, status: :not_found
+    end
+
+    def show
+      @encode_record = ActiveEncode::EncodeRecord.find(params[:id])
+      render json: @encode_record.raw_object, status: :ok
+    end
+  end
+end

data/app/jobs/active_encode/polling_job.rb

@@ -1,6 +1,6 @@
+# frozen_string_literal: true
 module ActiveEncode
   class PollingJob < ActiveJob::Base
-
     def perform(encode)
       encode.run_callbacks(:status_update) { encode }
       case encode.state

data/app/models/active_encode/encode_record.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 module ActiveEncode
   class EncodeRecord < ActiveRecord::Base
     # sql id, globalid, state, adapter, input filename/job title, timestamps

data/config/routes.rb

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+ActiveEncode::Engine.routes.draw do
+  resources :encode_record, only: [:show]
+end

data/db/migrate/20180822021048_create_active_encode_encode_records.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 class CreateActiveEncodeEncodeRecords < ActiveRecord::Migration[5.0]
   def change
     create_table :active_encode_encode_records do |t|

data/db/migrate/20190702153755_add_create_options_to_active_encode_encode_records.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+class AddCreateOptionsToActiveEncodeEncodeRecords < ActiveRecord::Migration[5.0]
+  def change
+    add_column :active_encode_encode_records, :create_options, :text
+  end
+end

data/db/migrate/20190712174821_add_progress_to_active_encode_encode_records.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+class AddProgressToActiveEncodeEncodeRecords < ActiveRecord::Migration[5.1]
+  def change
+    add_column :active_encode_encode_records, :progress, :float
+  end
+end

data/lib/active_encode.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 require 'active_encode/version'
 require 'active_encode/base'
 require 'active_encode/engine'

data/lib/active_encode/base.rb

@@ -1,5 +1,7 @@
+# frozen_string_literal: true
 require 'active_encode/core'
 require 'active_encode/engine_adapter'
+require 'active_encode/errors'
 require 'active_encode/status'
 require 'active_encode/technical_metadata'
 require 'active_encode/input'
@@ -17,6 +19,4 @@ module ActiveEncode #:nodoc:
     include Callbacks
     include GlobalID
   end
-
-  class NotFound < RuntimeError; end
 end

data/lib/active_encode/callbacks.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 require 'active_model/callbacks'
 
 module ActiveEncode

data/lib/active_encode/core.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 require 'active_support'
 require 'active_encode/callbacks'
 
@@ -44,19 +45,19 @@ module ActiveEncode
     end
 
     def initialize(input_url, options = nil)
-      @input = Input.new.tap{ |input| input.url = input_url }
+      @input = Input.new.tap { |input| input.url = input_url }
       @options = self.class.default_options(input_url).merge(Hash(options))
     end
 
     def create!
       run_callbacks :create do
-        merge!(self.class.engine_adapter.create(self.input.url, self.options))
+        merge!(self.class.engine_adapter.create(input.url, options))
       end
     end
 
     def cancel!
       run_callbacks :cancel do
-        merge!(self.class.engine_adapter.cancel(self.id))
+        merge!(self.class.engine_adapter.cancel(id))
       end
     end
 

data/lib/active_encode/engine.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 require 'rails'
 
 module ActiveEncode

data/lib/active_encode/engine_adapter.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 require 'active_encode/engine_adapters'
 require 'active_support/core_ext/class/attribute'
 require 'active_support/core_ext/string/inflections'

data/lib/active_encode/engine_adapters.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 module ActiveEncode
   # == Active Encode adapters
   #
@@ -13,8 +14,10 @@ module ActiveEncode
     autoload :ElasticTranscoderAdapter
     autoload :TestAdapter
     autoload :FfmpegAdapter
+    autoload :MediaConvertAdapter
+    autoload :PassThroughAdapter
 
-    ADAPTER = 'Adapter'.freeze
+    ADAPTER = 'Adapter'
     private_constant :ADAPTER
 
     class << self

data/lib/active_encode/engine_adapters/elastic_transcoder_adapter.rb

@@ -1,12 +1,27 @@
+# frozen_string_literal: true
+require 'addressable/uri'
+require 'aws-sdk-elastictranscoder'
+require 'file_locator'
+
 module ActiveEncode
   module EngineAdapters
     class ElasticTranscoderAdapter
-      # TODO: add a stub for an input helper (supplied by an initializer) that transforms encode.input into a zencoder accepted url
+      JOB_STATES = {
+        "Submitted" => :running, "Progressing" => :running, "Canceled" => :cancelled,
+        "Error" => :failed, "Complete" => :completed
+      }.freeze
+
+      # Require options to include :pipeline_id, :masterfile_bucket and :outputs
+      # Example :outputs value:
+      # [{ key: "quality-low/hls/fireworks", preset_id: '1494429796844-aza6zh', segment_duration: '2' },
+      #  { key: "quality-medium/hls/fireworks", preset_id: '1494429797061-kvg9ki', segment_duration: '2' },
+      #  { key: "quality-high/hls/fireworks", preset_id: '1494429797265-9xi831', segment_duration: '2' }]
       def create(input_url, options = {})
+        s3_key = copy_to_input_bucket input_url, options[:masterfile_bucket]
         job = client.create_job(
-          input: { key: input_url },
+          input: { key: s3_key },
           pipeline_id: options[:pipeline_id],
-          output_key_prefix: options[:output_key_prefix],
+          output_key_prefix: options[:output_key_prefix] || "#{SecureRandom.uuid}/",
           outputs: options[:outputs],
           user_metadata: options[:user_metadata]
         ).job
@@ -14,7 +29,7 @@ module ActiveEncode
         build_encode(job)
       end
 
-      def find(id, opts = {})
+      def find(id, _opts = {})
         build_encode(get_job_details(id))
       end
 
@@ -31,28 +46,33 @@ module ActiveEncode
           @client ||= Aws::ElasticTranscoder::Client.new
         end
 
+        def s3client
+          Aws::S3::Client.new
+        end
+
         def get_job_details(job_id)
-          client.read_job(id: job_id).job
+          client.read_job(id: job_id)&.job
         end
 
         def build_encode(job)
           return nil if job.nil?
-          encode = ActiveEncode::Base.new(convert_input(job), convert_options(job))
+          encode = ActiveEncode::Base.new(convert_input(job), {})
           encode.id = job.id
-          encode.state = convert_state(job)
-          encode.current_operations = convert_current_operations(job)
+          encode.state = JOB_STATES[job.status]
+          encode.current_operations = []
           encode.percent_complete = convert_percent_complete(job)
           encode.created_at = convert_time(job.timing["submit_time_millis"])
-          encode.updated_at = convert_time(job.timing["finish_time_millis"] || job.timing["start_time_millis"]) || encode.created_at
+          encode.updated_at = convert_time(job.timing["finish_time_millis"]) || convert_time(job.timing["start_time_millis"]) || encode.created_at
+
           encode.output = convert_output(job)
-          encode.errors = convert_errors(job)
+          encode.errors = job.outputs.select { |o| o.status == "Error" }.collect(&:status_detail).compact
 
-          encode.input.id = job.input.key
           tech_md = convert_tech_metadata(job.input.detected_properties)
-          [:width, :height, :frame_rate, :duration, :checksum, :audio_codec, :video_codec,
-           :audio_bitrate, :video_bitrate, :file_size].each do |field|
+          [:width, :height, :frame_rate, :duration, :file_size].each do |field|
             encode.input.send("#{field}=", tech_md[field])
           end
+
+          encode.input.id = job.id
           encode.input.state = encode.state
           encode.input.created_at = encode.created_at
           encode.input.updated_at = encode.updated_at
@@ -62,7 +82,12 @@ module ActiveEncode
 
         def convert_time(time_millis)
           return nil if time_millis.nil?
-          Time.at(time_millis / 1000)
+          Time.at(time_millis / 1000).utc
+        end
+
+        def convert_bitrate(rate)
+          return nil if rate.nil?
+          (rate.to_f * 1024).to_s
         end
 
         def convert_state(job)
@@ -78,13 +103,12 @@ module ActiveEncode
           end
         end
 
-        def convert_current_operations(_job)
-          current_ops = []
-          current_ops
+        def convert_percent_complete(job)
+          job.outputs.inject(0) { |sum, output| sum + output_percentage(output) } / job.outputs.length
         end
 
-        def convert_percent_complete(job)
-          case job.status
+        def output_percentage(output)
+          case output.status
           when "Submitted"
             10
           when "Progressing", "Canceled", "Error"
@@ -97,32 +121,72 @@ module ActiveEncode
         end
 
         def convert_input(job)
-          job.input
+          job.input.key
         end
 
-        def convert_options(_job_details)
-          {}
+        def copy_to_input_bucket(input_url, bucket)
+          case Addressable::URI.parse(input_url).scheme
+          when nil, 'file'
+            upload_to_s3 input_url, bucket
+          when 's3'
+            check_s3_bucket input_url, bucket
+          end
+        end
+
+        def check_s3_bucket(input_url, source_bucket)
+          # logger.info("Checking `#{input_url}'")
+          s3_object = FileLocator::S3File.new(input_url).object
+          if s3_object.bucket_name == source_bucket
+            # logger.info("Already in bucket `#{source_bucket}'")
+            s3_object.key
+          else
+            s3_key = File.join(SecureRandom.uuid, s3_object.key)
+            # logger.info("Copying to `#{source_bucket}/#{input_url}'")
+            target = Aws::S3::Object.new(bucket_name: source_bucket, key: input_url)
+            target.copy_from(s3_object, multipart_copy: s3_object.size > 15_728_640) # 15.megabytes
+            s3_key
+          end
+        end
+
+        def upload_to_s3(input_url, source_bucket)
+          # original_input = input_url
+          bucket = Aws::S3::Resource.new(client: s3client).bucket(source_bucket)
+          filename = FileLocator.new(input_url).location
+          s3_key = File.join(SecureRandom.uuid, File.basename(filename))
+          # logger.info("Copying `#{original_input}' to `#{source_bucket}/#{input_url}'")
+          obj = bucket.object(s3_key)
+          obj.upload_file filename
+
+          s3_key
+        end
+
+        def read_preset(id)
+          @presets ||= {}
+          @presets[id] ||= client.read_preset(id: id).preset
         end
 
         def convert_output(job)
-          job.outputs.collect do |o|
-            # It is assumed that the first part of the output key can be used to label the  output
-            # e.g. "quality-medium/somepath/filename.flv"
+          @pipeline ||= client.read_pipeline(id: job.pipeline_id).pipeline
+          job.outputs.collect do |joutput|
+            preset = read_preset(joutput.preset_id)
+            extension = preset.container == 'ts' ? '.m3u8' : ''
+            additional_metadata = {
+              managed: false,
+              id: joutput.id,
+              label: joutput.key.split("/", 2).first,
+              url: "s3://#{@pipeline.output_bucket}/#{job.output_key_prefix}#{joutput.key}#{extension}"
+            }
+            tech_md = convert_tech_metadata(joutput, preset).merge(additional_metadata)
+
             output = ActiveEncode::Output.new
-            output.id = o.id
-            output.label = o.key.split("/", 2).first
-            output.url = job.output_key_prefix + o.key
-            # TODO: If HLS is considered distinct from this output then it should be a different output
-            # TODO: If HLS is not considered distinct from this output then this should be handled by a method on a ActiveEncode::Base subclass or consuming client
-            # extras[:hls_url] = url + ".m3u8" if url.include?("/hls/") # TODO: find a better way to signal hls
-            tech_md = convert_tech_metadata(o)
+            output.state = convert_state(joutput)
+            output.created_at = convert_time(job.timing["submit_time_millis"])
+            output.updated_at = convert_time(job.timing["finish_time_millis"] || job.timing["start_time_millis"]) || output.created_at
+
             [:width, :height, :frame_rate, :duration, :checksum, :audio_codec, :video_codec,
-             :audio_bitrate, :video_bitrate, :file_size].each do |field|
+             :audio_bitrate, :video_bitrate, :file_size, :label, :url, :id].each do |field|
               output.send("#{field}=", tech_md[field])
             end
-            output.state = convert_state(o)
-            output.created_at = convert_time(job.timing["submit_time_millis"])
-            output.updated_at = convert_time(job.timing["finish_time_millis"] || job.timing["start_time_millis"]) || output.created_at
 
             output
           end
@@ -132,12 +196,13 @@ module ActiveEncode
           job.outputs.select { |o| o.status == "Error" }.collect(&:status_detail).compact
         end
 
-        def convert_tech_metadata(props)
-          return {} if props.blank?
+        def convert_tech_metadata(props, preset = nil)
+          return {} if props.nil? || props.empty?
           metadata_fields = {
             file_size: { key: :file_size, method: :itself },
             duration_millis: { key: :duration, method: :to_i },
             frame_rate: { key: :frame_rate, method: :to_i },
+            segment_duration: { key: :segment_duration, method: :itself },
             width: { key: :width, method: :itself },
             height: { key: :height, method: :itself }
           }
@@ -149,6 +214,19 @@ module ActiveEncode
             next if conversion.nil?
             metadata[conversion[:key]] = value.send(conversion[:method])
           end
+
+          unless preset.nil?
+            audio = preset.audio
+            video = preset.video
+            metadata.merge!(
+              audio_codec: audio&.codec,
+              audio_channels: audio&.channels,
+              audio_bitrate: convert_bitrate(audio&.bit_rate),
+              video_codec: video&.codec,
+              video_bitrate: convert_bitrate(video&.bit_rate)
+            )
+          end
+
           metadata
         end
     end