tootsie 0.9.0

data/.gitignore

@@ -0,0 +1,15 @@
+*~
+.DS_Store
+/.bundle
+/log
+/tmp
+/trash
+/config/development.yml
+/config/production.yml
+.rvmrc
+*.sublime-project
+*.sublime-workspace
+.rspec
+/coverage
+/pkg
+/Gemfile.lock

data/Gemfile

@@ -0,0 +1,2 @@
+source 'http://rubygems.org'
+gemspec

data/License

@@ -0,0 +1,7 @@
+Copyright © 2010, 2011 Alexander Staubo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,256 @@
+Tootsie
+=======
+
+Tootsie (formerly called Tranz) is a simple audio/video/image transcoding/modification application written in Ruby. It can transcode audio, video and images between different formats, and also perform basic manipulations such as scaling.
+
+Tootsie has the following external dependencies:
+
+* FFmpeg for transcoding of video and audio.
+* ImageMagick/GraphicsMagick for image conversion.
+* Amazon S3 for loading and storage of files (optional).
+* Amazon Simple Queue Service for internal task queue management (optional).
+
+Overview
+--------
+
+Tootsie is divided into multiple independent parts:
+
+* Job manager: finds new transcoding jobs and executes them.
+* FFmpeg, ImageMagick: performs the actual transcoding.
+* Queue: currently local file-based queues (for testing) and Amazon Simple Queue Service are supported.
+* Storage: currently web servers and Amazon S3 are supported.
+* Web service: A small RESTful API for managing jobs.
+
+The framework is designed to be easily pluggable, and to let you pick the parts you need to build a custom transcoding service. It is also designed to be easily distributed across many nodes.
+
+Execution flow
+--------------
+
+The task manager pops jobs from a queue and processes them. Each job specifies an input, an output, and transcoding parameters. Optionally the job may also specify a notification URL which is invoked to inform the caller about job progress.
+
+Supported inputs at the moment:
+
+* HTTP resource. Currently only public (non-authenticated) resources are supported.
+* Amazon S3 bucket resource. S3 buckets must have the appropriate ACLs so that Tootsie can read the files; if the input file is not public, Tootsie must be run with an AWS access key that is granted read access to the file.
+
+Supported outputs:
+
+* HTTP resource. The encoded file will be `POST`ed to a URL.
+* Amazon S3 bucket resource. Tootsie will need write permissions to any S3 buckets.
+
+Each job may have multiple outputs given a single input. Designwise, the reason for doing this -- as opposed to requiring that the client submit multiple jobs, one for each output -- is twofold:
+
+1. It allows the job to cache the input data locally for the duration of the job, rather than fetching it multiple times. One could suppose that multiple jobs could share the same cached input, but this would be awkward in a distributed setting where each node has its own file system; in such a case, a shared storage mechanism (file system, database or similar) would be needed.
+
+2. It allows the client to be informed when *all* transcoded versions are available, something which may drastically simplify client logic. For example, a web application submitting a job to produce multiple scaled versions of an image may only start showing these images when all versions have been produced. To know whether all versions have been produced, it needs to maintain state somewhere about the progress. Having a single job produce all versions means this state can be reduced to a single boolean value.
+
+When using multiple outputs per job one should keep in mind that this reduces job throughput, requiring more concurrent job workers to be deployed.
+
+FFmpeg and ImageMagick are invoked for each job to perform the transcoding. These are abstracted behind set of generic options specifying format, codecs, bit rate and so on.
+
+API
+===
+
+To schedule jobs, one uses the web service, a small app that supports job control methods:
+
+* POST `/job`: Schedule a job. Returns 201 if the job was created.
+* GET `/status`: Get current processing status as a JSON hash.
+
+The job must be posted as an JSON hash with the content type `application/json`. Common to all job scheduling POSTs are these keys:
+
+* `type`: Type of job. See sections below for details.
+* `notification_url`: Optional notification URL. Progress (including completion and failure) will be reported using POSTs.
+* `retries`: Maximum number of retries, if any. Defaults to 5.
+* `access_key`: Access key for calculating notification signature. See below.
+
+Job-specific parameters are provided in the key `params`.
+
+Access key
+----------
+
+(This part has not been written yet. Nor implemented, actually.)
+
+Notifications
+-------------
+
+If a notification URL is provided, events will be sent to it using `POST` requests as JSON data. These are 'fire and forget' and will currently not be retried on failure, and the response status code is ignored.
+
+There are several types of events, indicated by the `event` key:
+
+* `started`: The job was started.
+* `complete`: The job was complete. The key `time_taken` will contain the time taken for the job, in seconds. Additional data will be provided that are specific to the type of job.
+* `failed`: The job failed. The key `reason` will contain a textual explanation for the failure.
+* `failed_will_retry`: The job failed, but is being rescheduled for retrying. The key `reason` will contain a textual explanation for the failure.
+
+Video transcoding jobs
+----------------------
+
+Video jobs have the `type` key set to either `video`, `audio`. Currently, `audio` is simply an alias for `video` and handled by the same pipeline. The key `params` must be set to a hash with these keys:
+
+* `input_url`: URL to input file, either an HTTP URL or an S3 URL (see below).
+* `versions`: Either a hash or an array of such hashes, each with the following keys:
+  * `target_url`: URL to output resource, either an HTTP URL which accepts POSTs, or an S3 URL.
+  * `thumbnail`: If specified, a thumbnail will be generated based on the options in this hash with the following keys:
+    * `target_url`: URL to output resource, either an HTTP URL which accepts POSTs, or an S3 URL.
+    * `width`: Desired width of thumbnail, defaults to output width.
+    * `height`: Desired height of thumbnail, defaults to output height.
+    * `at_seconds`: Desired point (in seconds) at which the thumbnail frame should be captured. Defaults to 50% into stream.
+    * `at_fraction`: Desired point (in percentage) at which the thumbnail frame should be captured. Defaults to 50% into stream.
+    * `force_aspect_ratio`: If `true`, force aspect ratio; otherwise aspect is preserved when computing dimensions.
+  * `audio_sample_rate`: Audio sample rate, in hertz.
+  * `audio_bitrate`: Audio bitrate, in bits per second.
+  * `audio_codec`: Audio codec name, eg. `mp4`.
+  * `video_frame_rate`: video frame rate, in hertz.
+  * `video_bitrate`: video bitrate, in bits per second.
+  * `video_codec`: video codec name, eg. `mp4`.
+  * `width`: desired video frame width in pixels.
+  * `height`: desired video frame height in pixels.
+  * `format`: File format.
+  * `content_type`: Content type of resultant file. Tootsie will not be able to guess this at the moment.
+
+Completion notification provides the following data:
+
+* `outputs` contains an array of results. Each is a hash with the following keys:
+  * `url`: the completed file.
+  * `metadata`: image metadata as a hash. These are raw EXIF and IPTC data from ImageMagick.
+
+Image transcoding jobs
+----------------------
+
+Image jobs have the `type` key set to `image`. The key `params` must be set to a hash with these keys:
+
+* `input_url`: URL to input file, either an HTTP URL, `file:/path` URL or an S3 URL (see below).
+* `versions`: Either a hash or an array of such hashes, each with the following keys:
+  * `target_url`: URL to output resource, either an HTTP URL, `file:/path` URL which accepts POSTs, or an S3 URL.
+  * `width`: Optional desired width of output image.
+  * `height`: Optional desired height of output image.
+  * `scale`: One of the following values:
+    * `down` (default): The input image is scaled to fit within the dimensions `width` x `height`. If only `width` or only `height` is specified, then the other component will be computed from the aspect ratio of the input image.
+    * `up`: As `within`, but allow scaling to dimensions that are larger than the input image.
+    * `fit`: Similar to `down`, but the dimensions are chosen so the output width and height are always met or exceeded. In other words, if you pass in an image that is 100x50, specifying output dimensions as 100x100, then the output image will be 150x100.
+    * `none`: Don't scale at all.
+  * `crop`: If true, crop the image to the output dimensions.
+  * `format`: Either `jpeg`, `png` or `gif`.
+  * `quality`: A quality value between 0.0 and 1.0 which will be translated to a compression level depending on the output coding. The default is 1.0.
+  * `strip_metadata`: If true, metadata such as EXIF and IPTC will be deleted. For thumbnails, this often reduces the file size considerably.
+  * `medium`: If `web`, the image will be optimized for web usage. See below for details.
+  * `content_type`: Content type of resultant file. The system will be able to guess basic types such as `image/jpeg`.
+
+Note that scaling always preserves the aspect ratio of the original image; in other words, if the original is 100 x 200, then passing the dimensions 100x100 will produce an image that is 50x100. Enabling cropping, however, will force the aspect ratio of the specified dimensions.
+
+If the option `medium` specifies `web`, the following additional transformations will be performed:
+
+* The image will be automatically rotated based on EXIF orientation metadata, since web browsers don't do this.
+* CMYK images will be converted to RGB, since most web browsers don't seem to display CMYK correctly.
+
+Completion notification provides the following data:
+
+* `outputs` contains an array of results. Each is a hash with the following keys:
+  * `url`: URL for the completed file.
+* `metadata`: image metadata as a hash. These are raw EXIF and IPTC data from ImageMagick.
+* `width`: width, in pixels, of original image.
+* `height`: height, in pixels, of original image.
+* `depth`: depth, in bits, of original image.
+
+Note about S3 URLs
+------------------
+
+To specify S3 URLs, we use a custom URI format:
+
+    s3:<bucketname></path/to/file>[?<options>]
+
+The components are:
+
+* bucketname: The name of the S3 bucket.
+* /path/to/file: The actual S3 key.
+* options: Optional parameters for storage, an URL query string.
+
+The options are:
+
+* `acl`: One of `private` (default), `public-read`, `public-read-write` or `authenticated-read`.
+* `storage_class`: Either `standard` (default) or `reduced_redundancy`.
+* `content_type`: Override stored content type.
+
+Example S3 URLs:
+
+* `s3:myapp/video`
+* `s3:myapp/thumbnails?acl=public-read&storage_class=reduced_redundancy`
+* `s3:myapp/images/12345?content_type=image/jpeg`
+
+Current limitations
+===================
+
+* Daemon supports only one task manager thread at a time.
+* Transcoding options are very basic.
+* No client access control; anyone can submit jobs.
+
+Requirements
+============
+
+* Ruby 1.9.1 or later.
+* Bundler.
+
+For video jobs:
+
+* FFmpeg
+
+For image jobs:
+
+* ImageMagick
+* Exiv2
+* pngcrush (optional)
+
+Installation
+============
+
+* Fetch Git repositroy: `git clone git@github.com:origo/tootsie.git`.
+* Install Bundler with `gem install bundler`.
+* Install dependencies with `cd tootsie; bundle install`.
+
+Running
+=======
+
+Create a configuration under `config`, eg. `config/development.yml`:
+
+    --- 
+      aws_access_key_id: <your Amazon key>
+      aws_secret_access_key: <your Amazon secret>
+      sqs_queue_name: tootsie
+
+Start the task manager with `bin/tootsie_task_manager start`. You can specify the number of workers with `-n`.
+
+To run the web service, you will need a Rack-compatible web server, such as Unicorn or Thin. To start with Thin on port 9090:
+
+    $ thin --daemonize --rackup config.ru --port 9090 start
+
+Jobs may now be posted to the web service API. For example:
+
+    $ cat << END | curl -d @- http://localhost:9090/job
+    {
+      'type': 'video',
+      'notification_url': 'http://example.com/transcoder_notification',
+      'params': {
+        'input_url': 'http://example.com/test.3gp',
+        'outputs': {
+          'target_url': 's3:mybucket/test.mp4?acl=public_read',
+          'audio_sample_rate': 44100,
+          'audio_bitrate': 64000,
+          'format': 'flv',
+          'content_type': 'video/x-flv'
+        }
+      }
+    }
+    END
+
+License
+=======
+
+This software is licensed under the MIT License.
+
+Copyright © 2010, 2011 Alexander Staubo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/Tootsie.gemspec

@@ -0,0 +1,36 @@
+# -*- encoding: utf-8 -*-
+
+$:.push File.expand_path("../lib", __FILE__)
+require "tootsie/version"
+
+Gem::Specification.new do |s|
+  s.name        = "tootsie"
+  s.version     = Tootsie::VERSION
+  s.authors     = ["Alexander Staubo"]
+  s.email       = ["alex@origo.no"]
+  s.homepage    = "http://github.com/alexstaubo/tootsie"
+  s.summary     = s.description = %{Tootsie is a simple audio/video/image transcoding/modification application.}
+
+  s.rubyforge_project = "tootsie"
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  s.add_runtime_dependency 'json', ['~> 1.4.6']
+  s.add_runtime_dependency 'sinatra', ['~> 1.0']
+  s.add_runtime_dependency 'activesupport', ['~>3.0.0']
+  s.add_runtime_dependency 'httpclient', ['~>2.2.1']
+  s.add_runtime_dependency 'builder', ['~> 2.1.2']
+  s.add_runtime_dependency 'mime-types', ['~> 1.16']
+  s.add_runtime_dependency 'xml-simple', ['~> 1.0.12']
+  s.add_runtime_dependency 'thin', ['~> 1.2.7']
+  s.add_runtime_dependency 's3', ['~> 0.3.7']
+  s.add_runtime_dependency 'sqs', ['~> 0.1.2']
+  s.add_runtime_dependency 'unicorn', ['~> 4.1.1']
+  s.add_runtime_dependency 'i18n', ['>= 0.4.2']
+  s.add_runtime_dependency 'scashin133-syslog_logger', ['~> 1.7.3']
+  s.add_development_dependency "rspec"
+  s.add_development_dependency "rake"
+end

data/bin/tootsie_task_manager

@@ -0,0 +1,82 @@
+#!/usr/bin/env ruby
+
+ENV['BUNDLE_GEMFILE'] = File.expand_path('../../Gemfile', __FILE__)
+
+require 'rubygems'
+begin
+  require 'bundler'
+rescue LoadError
+  # Ignore this
+else
+  Bundler.setup
+end
+
+$:.unshift(File.expand_path('../../lib', __FILE__))
+require 'tootsie'
+
+environment = ENV['RACK_ENV']
+environment ||= :development
+
+num_workers = 4
+logger = nil
+
+ARGV.options do |opts|
+  opts.banner = "Usage: #{File.basename($0)} [OPTIONS] [start | stop | restart | status]"
+  opts.separator ""
+  opts.on("-e", "--environment=env", String, 
+    "Environment to run in (default: #{environment})") do |value| 
+    environment = value
+  end
+  opts.on("-n", "--num-workers=WORKERS", Integer,
+    "Specify number of workers to fork (defaults to #{num_workers}.") do |value|
+    num_workers = [1, value.to_i].max
+  end
+  opts.on("-l TARGET", "--logger TARGET", String,
+    "Log to TARGET, which is either a file name or 'syslog'.") do |value|
+    if value == 'syslog'
+      require 'syslog_logger'
+      logger = SyslogLogger.new('tootsie')
+    else
+      logger = Logger.new(value)
+    end
+  end
+  opts.on("-h", "--help", "Show this help message.") do
+    puts opts
+    exit
+  end
+  opts.parse!
+  if ARGV.empty?
+    puts "Nothing to do. Run with -h for help."
+    exit
+  end
+end
+
+controller = Tootsie::Daemon.new(
+  :root => File.join(File.dirname(__FILE__), "/.."),
+  :pid_file => File.join(File.dirname(__FILE__), "/../tmp/task_manager.pid"),
+  :logger => logger)
+
+spawner = Spawner.new(:num_children => num_workers, :logger => controller.logger)
+
+controller.on_spawn do
+  $0 = "tootsie: master"
+  spawner.on_spawn do
+    $0 = "tootsie: worker"
+    Signal.trap('TERM') do
+      exit(2)
+    end
+    app = Tootsie::Application.new(
+      :environment => environment,
+      :logger => controller.logger)
+    app.configure!
+    begin
+      app.task_manager.run!
+    rescue SystemExit, Interrupt
+    end
+  end
+  spawner.run
+end
+controller.on_terminate do
+  spawner.terminate
+end
+controller.control(ARGV)

data/config.ru

@@ -0,0 +1,22 @@
+require 'rubygems'
+require 'bundler'
+Bundler.setup
+Bundler.require
+
+$:.unshift(File.join(File.dirname(__FILE__), "/lib"))
+require 'tootsie'
+
+environment = ENV['RACK_ENV'] ||= 'development'
+
+app = Tootsie::Application.new(
+  :environment => environment,
+  :logger => ENV["rack.logger"])
+app.configure!
+
+if environment == 'development'
+  Thread.new do
+    Tootsie::Application.get.task_manager.run!
+  end
+end
+
+run Tootsie::WebService

data/config/development-sample.yml

@@ -0,0 +1,4 @@
+---
+  aws_access_key_id: <your_key>
+  aws_secret_access_key: <your_secret>
+  sqs_queue_name: tootsie

data/lib/tootsie.rb

@@ -0,0 +1,21 @@
+require 'active_support/core_ext/hash'
+
+require 'tootsie/application'
+require 'tootsie/client'
+require 'tootsie/configuration'
+require 'tootsie/command_runner'
+require 'tootsie/daemon'
+require 'tootsie/spawner'
+require 'tootsie/ffmpeg_adapter'
+require 'tootsie/image_metadata_extractor'
+require 'tootsie/input'
+require 'tootsie/task_manager'
+require 'tootsie/tasks/job_task'
+require 'tootsie/tasks/notify_task'
+require 'tootsie/output'
+require 'tootsie/web_service'
+require 'tootsie/processors/video_processor'
+require 'tootsie/processors/image_processor'
+require 'tootsie/queues/sqs_queue'
+require 'tootsie/queues/file_system_queue'
+require 'tootsie/s3_utilities'