viddl-rb 0.68 → 0.70

data/Gemfile.lock

@@ -1,19 +1,38 @@
 PATH
   remote: .
   specs:
-    viddl-rb (0.61)
+    viddl-rb (0.68)
+      jruby-openssl
+      mechanize
       nokogiri
 
 GEM
   remote: http://rubygems.org/
   specs:
-    mime-types (1.18)
-    minitest (2.12.1)
-    nokogiri (1.5.2)
-    nokogiri (1.5.2-java)
+    bouncy-castle-java (1.5.0146.1)
+    domain_name (0.5.3)
+      unf (~> 0.0.3)
+    jruby-openssl (0.7.7)
+      bouncy-castle-java (>= 1.5.0146.1)
+    mechanize (2.5.1)
+      domain_name (~> 0.5, >= 0.5.1)
+      mime-types (~> 1.17, >= 1.17.2)
+      net-http-digest_auth (~> 1.1, >= 1.1.1)
+      net-http-persistent (~> 2.5, >= 2.5.2)
+      nokogiri (~> 1.4)
+      ntlm-http (~> 0.1, >= 0.1.1)
+      webrobots (~> 0.0, >= 0.0.9)
+    mime-types (1.19)
+    minitest (3.3.0)
+    net-http-digest_auth (1.2.1)
+    net-http-persistent (2.7)
+    nokogiri (1.5.5-java)
+    ntlm-http (0.1.1)
     rake (0.9.2.2)
     rest-client (1.6.7)
       mime-types (>= 1.16)
+    unf (0.0.5-java)
+    webrobots (0.0.13)
 
 PLATFORMS
   java

data/README.md

@@ -1,49 +1,110 @@
-__viddl-rb:__  
-Created by Marc Seeger (@rb2k)  
-Repo: http://github.com/rb2k/viddl-rb  
-[![Build Status](https://secure.travis-ci.org/rb2k/viddl-rb.png)](http://travis-ci.org/rb2k/viddl-rb) [![Dependency Status](https://gemnasium.com/rb2k/viddl-rb.png)](https://gemnasium.com/rb2k/viddl-rb)
-
-
-
-__Installation:__  
-gem install viddl-rb
-
-__Usage:__  
-
-Download a video:  
-    viddl-rb http://www.youtube.com/watch?v=QH2-TGUlwu4
-
-Download a video and extract the audio:  
-    viddl-rb http://www.youtube.com/watch?v=QH2-TGUlwu4 --extract-audio 
-
-In both cases we'll name the output file according to the video title.
-
-__Youtube plugin specifics:__ 
-
-Download all videos on a playlist:  
-    viddl-rb http://www.youtube.com/playlist?list=PL7E8DA0A515924126
-
-Download all videos from a user:  
-    viddl-rb http://www.youtube.com/user/tedtalksdirector
-
-Filter videos to download from a user/playlist:  
-    viddl-rb http://www.youtube.com/user/tedtalksdirector --filter=internet/i
-
-The --filter argument accepts a regular expression and will only download videos where the title matches the regex.
-The /i option does a case-insensitive search.
-
-__Requirements:__
-
-* curl/wget or the [progress bar](http://github.com/nex3/ruby-progressbar/) gem  
-* [Nokogiri](http://nokogiri.org/)
-* [Mechanize](http://mechanize.rubyforge.org/)
-* ffmpeg if you want to extract audio tracks from the videos
-
-
-__Contributors:__
-
-* [kl](https://github.com/kl): Windows support (who knew!), bug fixes, veoh plugin, metacafe plugin   
-* [divout](https://github.com/divout) aka Ivan K: blip.tv plugin, bugfixes
-* Sniper: bugfixes
-* [Serabe](https://github.com/Serabe) aka Sergio Arbeo: packaging viddl as a binary
-* [laserlemon](https://github.com/laserlemon): Adding gemnasium images to readme
+__viddl-rb:__  
+Initially created by Marc Seeger (@rb2k)  
+Repo: http://github.com/rb2k/viddl-rb  
+[![Build Status](https://secure.travis-ci.org/rb2k/viddl-rb.png)](http://travis-ci.org/rb2k/viddl-rb) [![Dependency Status](https://gemnasium.com/rb2k/viddl-rb.png)](https://gemnasium.com/rb2k/viddl-rb)
+
+__Installation:__
+
+gem install viddl-rb
+
+__Usage:__  
+
+Download a video:  
+    viddl-rb http://www.youtube.com/watch?v=QH2-TGUlwu4
+
+Download a video and extract the audio:  
+    viddl-rb http://www.youtube.com/watch?v=QH2-TGUlwu4 --extract-audio 
+
+In both cases we'll name the output file according to the video title.
+
+Setting the video save directory:  
+    viddl-rb http://vimeo.com/38372260 --save-dir=C:/myvideos
+
+The --save-dir option works with both absolute and relative paths (relative based on the directory viddl-rb is run from).
+If you want to save to a folder with spaces in it, you have to quote the path like this: --save-dir="C:/my videos"
+
+__Youtube plugin specifics:__  
+
+Download all videos on a playlist:  
+    viddl-rb http://www.youtube.com/playlist?list=PL7E8DA0A515924126
+
+Download all videos from a user:  
+    viddl-rb http://www.youtube.com/user/tedtalksdirector
+
+Filter videos to download from a user/playlist:  
+    viddl-rb http://www.youtube.com/user/tedtalksdirector --filter=internet/i
+
+The --filter argument accepts a regular expression and will only download videos where the title matches the regex.
+The /i option does a case-insensitive search.
+
+__Library Usage:__
+
+```ruby
+require 'viddl-rb'
+
+download_urls = ViddlRb.get_urls("http://www.youtube.com/watch?v=QH2-TGUlwu4")
+download_urls.first     # => "http://o-o.preferred.arn06s04.v3.lscac ..."
+```
+
+The ViddlRb module has the following module public methods:
+
+* __get_urls_names(url)__
+-- Returns an array of one or more hashes that has the keys :url which
+points to the download url and :name which points to the name 
+(which is a filename safe version of the video title with a file extension).
+Returns nil if the url is not recognized by any plugins.
+
+* __get_urls_exts(url)__
+-- Same as get_urls_names but with just the file extension (for example ".mp4")
+instead of the full filename, and the :name key is replaced with :ext.
+Returns nil if the url is not recognized by any plugins.
+
+* __get_urls(url)__
+-- Returns an array of download urls for the specified video url.
+Returns nil if the url is not recognized by any plugins.
+
+* __get_names(url)__
+-- Returns an array of filenames for the specified video url.
+Returns nil if the url is not recognized by any plugins.
+
+* __io=(io_object)__
+-- By default all plugin output to stdout will be suppressed when the library is used.
+If you are interested in the output of a plugin, you can set an IO object that
+will receive all plugin output using this method. For example:
+
+```ruby
+require 'viddl-rb'
+
+ViddlRb.io = $stdout     # plugins will now write their output to $stdout
+```
+
+All the __get__ methods in the ViddlRb module will raise either a ViddlRb::PluginError or a ViddlRb::DownloadError if the plugin fails. 
+A ViddlRb::PluginError is raised if the plugin fails in an unexpected way, and a ViddlRb::DownloadError is raised if the video could not be downloaded for some reason. 
+An example of that is if a Youtube video is not embeddable - then it can't be downloaded.
+
+```ruby
+begin
+  ViddlRb.get_urls(video_url)
+rescue ViddlRb::DownloadError => e
+  puts "Could not get download url: #{e.message}"
+rescue ViddlRb::PluginError => e
+  puts "Plugin blew up! #{e.message}\n" +
+       "Backtrace:\n#{e.backtrace.join("\n")}"
+end
+```
+
+__Requirements:__
+
+* curl/wget or the [progress bar](http://github.com/nex3/ruby-progressbar/) gem  
+* [Nokogiri](http://nokogiri.org/)
+* [Mechanize](http://mechanize.rubyforge.org/)
+* ffmpeg if you want to extract audio tracks from the videos
+
+__Co Maintainer:__
+* [kl](https://github.com/kl): Windows support (who knew!), bug fixes, veoh plugin, metacafe plugin, refactoring it into a library, ...
+
+__Contributors:__  
+* [divout](https://github.com/divout) aka Ivan K: blip.tv plugin, bugfixes
+* Sniper: bugfixes
+* [Serabe](https://github.com/Serabe) aka Sergio Arbeo: packaging viddl as a binary
+* [laserlemon](https://github.com/laserlemon): Adding gemnasium images to readme

data/Rakefile

@@ -1,8 +1,18 @@
-require 'rubygems'
-require 'rake/testtask'
-
-task :default => [:test]
-
-Rake::TestTask.new do |t|
-  t.pattern = "spec/*_spec.rb"
-end
+require 'rubygems'
+require 'bundler/setup'
+require 'rake/testtask'
+
+task :default => [:test]
+
+Rake::TestTask.new(:test) do |t|
+  #t.pattern = "spec/*_spec.rb"
+  t.test_files = ["spec/lib_spec.rb", "spec/url_extraction_spec.rb"]
+end
+
+Rake::TestTask.new(:test_lib) do |t|
+  t.test_files = FileList["spec/lib_spec.rb"]
+end
+
+Rake::TestTask.new(:test_extract) do |t|
+  t.test_files = FileList["spec/url_extraction_spec.rb"]
+end

data/TODO.txt

@@ -0,0 +1,3 @@
+* wrap all classes used by the lib in a module (for namespace reasons)
+* add save_file method to library
+

data/bin/helper/downloader.rb

@@ -0,0 +1,25 @@
+
+# Downloader iterates over a download queue and downloads and saves each video in the queue.
+class Downloader
+  class DownloadFailedError < StandardError; end
+
+  def download(download_queue, params)
+    download_queue.each do |url_name|
+      url = url_name[:url]
+      name = url_name[:name]
+
+      result = save_file(url, name, params[:save_dir])
+      unless result
+        raise DownloadFailedError, "Download for #{name} failed."
+      else
+        puts "Download for #{name} successful."
+        AudioHelper.extract(name) if params[:extract_audio]
+      end
+    end
+  end
+
+  # TODO save_dir is not used yet
+  def save_file(url, name, save_dir)
+    ViddlRb::DownloadHelper.save_file(url, name, save_dir)
+  end
+end

data/bin/helper/driver.rb

@@ -0,0 +1,47 @@
+
+# The Driver class drives the application logic in the viddl-rb utility.
+# It gets the correct plugin for the given url and passes a download queue
+# (that it gets from the plugin) to the Downloader object which downloads the videos.
+class Driver
+
+  def initialize(param_hash)
+    @params = param_hash
+    @downloader = Downloader.new
+  end
+
+  #starts the downloading process or print just the urls or names.
+  def start
+    queue = get_download_queue
+
+    if @params[:url_only]
+      queue.each { |url_name| puts url_name[:url] }
+    elsif @params[:title_only]
+      queue.each { |url_name| puts url_name[:name] }
+    else
+      @downloader.download(queue, @params)
+    end
+  end
+
+  private
+
+  #finds the right plugins and returns the download queue.
+  def get_download_queue
+    url = @params[:url]
+    plugin = ViddlRb::PluginBase.registered_plugins.find { |p| p.matches_provider?(url) }
+    raise "ERROR: No plugin seems to feel responsible for this URL." unless plugin
+    puts "Using plugin: #{plugin}"
+
+    begin
+      #we'll end up with an array of hashes with they keys :url and :name 
+      plugin.get_urls_and_filenames(url, @params)
+      
+    rescue ViddlRb::PluginBase::CouldNotDownloadVideoError => e
+      raise "ERROR: The video could not be downloaded.\n" +
+            "Reason: #{e.message}"
+    rescue StandardError => e
+      raise "Error while running the #{plugin.name.inspect} plugin. Maybe it has to be updated?\n" +
+            "Error: #{e.message}.\n" +
+            "Backtrace:\n#{e.backtrace.join("\n")}"  
+    end
+  end
+end

data/bin/helper/parameter-parser.rb

@@ -0,0 +1,67 @@
+
+# ParameterParser parses the program parameters.
+# If the parameters are not valid in some way an exception is raised.
+class ParameterParser
+
+  DEFAULT_SAVE_DIR = "."
+
+  #returns a hash with the parameters in it:
+  # :url            => the video url
+  # :extract_audio  => should attempt to extract audio? (true/false)
+  # :url_only       => do not download, only print the urls to stdout
+  # :title_only     => do not download, only print the titles to stdout
+  # :youtube_filer  => a regular expression used ot fileter youtube playlists
+  # :save_dir       => the directory where the videos are saved
+  def self.parse_app_parameters
+    check_valid_parameters!
+
+    params = {}
+    params[:url] = ARGV.first
+    params[:extract_audio] = ARGV.include?("--extract-audio")
+    params[:url_only] = ARGV.include?("--url-only")
+    params[:title_only] = ARGV.include?("--title-only")
+    params[:playlist_filter] = get_youtube_filter
+    params[:save_dir] = get_save_dir  
+    params
+  end
+
+  #check if parameters are valid.
+  #the exceptions raised by this method are caught by the viddl-rb bin utility.
+  def self.check_valid_parameters!
+    if ARGV.empty?
+      raise "Usage: viddl-rb URL [--extract-audio]"
+    elsif !ARGV.first.match(/^http/)
+      raise "ERROR: Please include 'http' with your URL e.g. http://www.youtube.com/watch?v=QH2-TGUlwu4"
+    elsif ARGV.include?("--extract-audio") && !DownloadHelper.os_has?("ffmpeg")
+      raise "ERROR: To extract audio you need to have ffmpeg on your system"
+    end
+  end
+
+  #gets the regular expression used to filter youtube playlists.
+  def self.get_youtube_filter
+    filter = ARGV.find { |arg| arg =~ /--filter=./ }  # --filter= and at least one more character
+    return nil unless filter
+
+    ignore_case = filter.include?("/i")
+    regex = filter[/--filter=(.*?)(?:\/|$)/, 1]   # everything up to the first / (could be an empty string)
+    raise "ERROR: '#{regex}' is not a valid regular expression" unless is_valid_regex?(regex)
+    Regexp.new(regex, ignore_case)
+  end
+
+  #checks if the string is a valid regex (for example "*****" is not)
+  def self.is_valid_regex?(regex)
+    Regexp.compile(regex)
+  rescue RegexpError
+    false
+  end
+  
+  #gets the directory used for saving videos in.
+  def self.get_save_dir
+    save_dir = ARGV.find { |arg| arg =~ /--save-dir=./ }
+    return DEFAULT_SAVE_DIR unless save_dir
+
+    dir = save_dir[/--save-dir=(.+)/, 1]
+    raise "ERROR: '#{dir}' is not a valid directory or does not exist" unless File.directory?(dir)
+    dir
+  end
+end

data/bin/viddl-rb

@@ -1,118 +1,39 @@
-#!/usr/bin/env ruby
-$LOAD_PATH << File.join(File.dirname(__FILE__), '..', 'helper')
-
-require "rubygems"
-require "nokogiri"
-require "mechanize"
-require "cgi"
-require "open-uri"
-require "open3"
-require "download-helper.rb"
-require "plugin-helper.rb"
-
-if ARGV[0].nil?
-	puts "Usage: viddl-rb URL [--extract-audio]"
-	exit
-end
-
-puts "Loading Plugins"
-Dir[File.join(File.dirname(__FILE__),"../plugins/*.rb")].each do |plugin|
-	load plugin
-end
-
-puts "Plugins loaded: #{PluginBase.registered_plugins.inspect}"
-
-url = ARGV[0]
-extract_audio = ARGV.include?('--extract-audio')
-url_only = ARGV.include?('--url-only')
-title_only = ARGV.include?('--title-only')
-
-puts "Will try to extract audio: #{extract_audio}."
-
-unless url.match(/^http/)
-	puts "Please include 'http' with your URL e.g. http://www.youtube.com/watch?v=QH2-TGUlwu4"
-	exit(1)
-end
-
-puts "Analyzing URL: #{url}"
-#Check all plugins for a match
-PluginBase.registered_plugins.each do |plugin|
-	if plugin.matches_provider?(url)
-		puts "#{plugin}: true"
-		begin
-			#we'll end up with an array of hashes with they keys :url and :name 
-			download_queue = plugin.get_urls_and_filenames(url)
-		rescue StandardError => e
-			puts "Error while running the #{plugin.name.inspect} plugin. Maybe it has to be updated? Error: #{e.message}."	
-			exit(1)
-		end
-
-		if url_only
-			download_queue.each{|url_name| puts url_name[:url]}
-			exit
-		elsif title_only
-			download_queue.each{|url_name| puts url_name[:name]}
-			exit
-		end
-
-		download_queue.each do |url_name|
-			result = DownloadHelper.save_file(url_name[:url], url_name[:name])
-			if result
-				puts "Download for #{url_name[:name]} successful."
-				if extract_audio
-					puts "Extracting audio for #{url_name[:name]}"
-					if DownloadHelper.os_has?('ffmpeg')
-						no_ext_filename = url_name[:name].split('.')[0..-1][0]
-						#capture stderr because ffmpeg expects an output param and will error out
-						puts "Gathering information about the downloaded file."
-						file_info = Open3.popen3("ffmpeg -i #{url_name[:name]}") {|stdin, stdout, stderr, wait_thr| stderr.read }
-						puts "Done gathering information about the downloaded file."
-						if !file_info.to_s.empty?
-						  audio_format_matches = file_info.match(/Audio: (\w*)/)
-						  if audio_format_matches
-						  	audio_format = audio_format_matches[1]
-						  	puts "detected audio format: #{audio_format}"
-						  else
-						  	puts "Couldn't find any audio:\n#{file_info.inspect}"
-						  	next
-						  end
-						  
-						  extension_mapper = {
-							'aac' => 'm4a',
-							'mp3' => 'mp3',
-							'vorbis' => 'ogg'
-						  }
-
-						  if extension_mapper.key?(audio_format)
-						  	output_extension = extension_mapper[audio_format]
-						  else
-							#lame fallback
-							puts "Unknown audio format: #{audio_format}, using name as extension: '.#{audio_format}'."
-							output_extension = audio_format
-						  end
-						  output_filename = "#{no_ext_filename}.#{output_extension}"
-						  if File.exist?(output_filename)
-						  	puts "Audio file seems to exist already, removing it before extraction."
-						    File.delete(output_filename)
-						  end
-						  Open3.popen3("ffmpeg -i #{url_name[:name]} -vn -acodec copy #{output_filename}") {|stdin, stdout, stderr, wait_thr| stdout.read }
-						  puts "Done extracting audio to #{output_filename}"
-						else
-						  puts "Error while checking audio track of #{url_name[:name]}"
-						end
-					else
-						puts "Didn't detect ffmpeg on your system, can't extract audio."
-					end
-				end
-			else
-				puts "Download for #{url_name[:name]} failed."
-			end
-		end
-		#plugin matched and downloaded, we're done
-		exit
-	else
-		puts "#{plugin}: false"
-	end
-end
-
-puts "No plugin seems to feel responsible for this URL."
+#!/usr/bin/env ruby
+$LOAD_PATH << File.join(File.dirname(__FILE__), '..', 'helper')  # general helpers
+$LOAD_PATH << File.join(File.dirname(__FILE__), 'helper')  	     # bin-specific helpers
+
+require "rubygems"
+require "nokogiri"
+require "mechanize"
+require "cgi"
+require "open-uri"
+require "open3"
+
+require "driver.rb"
+require "downloader.rb"
+require "download-helper.rb"
+require "parameter-parser.rb"
+require "plugin-helper.rb"
+require "audio-helper.rb"
+require "utility-helper.rb"
+
+begin
+  # params is a hash with keys for each of the parameters passed in.
+  # see helper/parameter-parser.rb for what those keys are.
+  params = ParameterParser.parse_app_parameters
+
+  puts "Loading Plugins"
+  ViddlRb::UtilityHelper.load_plugins
+  "Plugins loaded: #{ViddlRb::PluginBase.registered_plugins.inspect}"
+
+  puts "Will try to extract audio: #{params[:extract_audio] == true}."
+  puts "Analyzing URL: #{params[:url]}"
+
+  app = Driver.new(params)   
+  app.start                  # starts the download process
+
+rescue StandardError => e
+  puts "#{e.message}"
+  exit(1)
+end
+