viddl-rb 0.8 → 0.61

data/CHANGELOG.txt

@@ -0,0 +1,14 @@
+=== 0.6
+* Soundcloud support
+* Removed megavideo plugin
+
+=== 0.4.x
+* New Features
+  * Use wget or curl (if present)
+
+* Bugfixes
+  * Fixed Vimeo Plugin
+
+=== 0.3
+* New Features
+  * added megavideo plugin

data/Gemfile.lock

@@ -1,43 +1,21 @@
 PATH
   remote: .
   specs:
-    viddl-rb (0.79)
-      mechanize
-      nokogiri (~> 1.5.0)
-      progressbar
+    viddl-rb (0.6)
+      nokogiri
 
 GEM
   remote: http://rubygems.org/
   specs:
-    domain_name (0.5.12)
-      unf (>= 0.0.5, < 1.0.0)
-    http-cookie (1.0.1)
-      domain_name (~> 0.5)
-    mechanize (2.7.1)
-      domain_name (~> 0.5, >= 0.5.1)
-      http-cookie (~> 1.0.0)
-      mime-types (~> 1.17, >= 1.17.2)
-      net-http-digest_auth (~> 1.1, >= 1.1.1)
-      net-http-persistent (~> 2.5, >= 2.5.2)
-      nokogiri (~> 1.4)
-      ntlm-http (~> 0.1, >= 0.1.1)
-      webrobots (>= 0.0.9, < 0.2)
-    mime-types (1.23)
-    minitest (5.0.4)
-    net-http-digest_auth (1.3)
-    net-http-persistent (2.8)
-    nokogiri (1.5.10)
-    ntlm-http (0.1.1)
-    progressbar (0.20.0)
-    rake (10.0.4)
+    mime-types (1.17.2)
+    minitest (2.11.3)
+    nokogiri (1.5.0)
+    rake (0.9.2.2)
     rest-client (1.6.7)
       mime-types (>= 1.16)
-    unf (0.1.1)
-      unf_ext
-    unf_ext (0.0.6)
-    webrobots (0.1.1)
 
 PLATFORMS
+  java
   ruby
 
 DEPENDENCIES

data/README.md

@@ -1,122 +1,34 @@
-__viddl-rb:__ 
-Initially created by Marc Seeger (@rb2k)  
-Repo: http://github.com/rb2k/viddl-rb  
-[![Gem Version](https://badge.fury.io/rb/viddl-rb.png)](http://badge.fury.io/rb/viddl-rb)[![Build Status](https://secure.travis-ci.org/rb2k/viddl-rb.png)](http://travis-ci.org/rb2k/viddl-rb) [![Dependency Status](https://gemnasium.com/rb2k/viddl-rb.png)](https://gemnasium.com/rb2k/viddl-rb)
-
-__Installation:__
-
-gem install viddl-rb
-
-__Usage:__
-
-Download a video:  
-    ```viddl-rb http://www.youtube.com/watch?v=QH2-TGUlwu4```
-
-Viddl-rb supports the following command line options:
-```
--e, --extract-audio              Save video audio to file
--u, --url-only                   Prints url without downloading
--t, --title-only                 Prints title without downloading
--f, --filter REGEX               Filters a video playlist according to the regex (Youtube only right now)
--s, --save-dir DIRECTORY         Specifies the directory where videos should be saved
--d, --downloader TOOL            Specifies the tool to download with. Supports 'wget', 'curl' and 'net-http'
--q, --quality QUALITY            Specifies the video format and resolution in the following way => resolution:extension (e.g. 720:mp4). Currently only supported by the Youtube plugin.
--h, --help                       Displays the help screen
-```
-
-Download a video and extract the audio:  
-    ```viddl-rb http://www.youtube.com/watch?v=QH2-TGUlwu4 --extract-audio```
-
-In both cases we'll name the output file according to the video title.
-
-Setting the video save directory:  
-    ```viddl-rb http://vimeo.com/38372260 --save-dir C:/myvideos```
-
-The --save-dir option works with both absolute and relative paths (relative based on the directory viddl-rb is run from).
-If you want to save to a folder with spaces in it, you have to quote the path like this: --save-dir "C:/my videos"
-
-__Youtube plugin specifics:__  
-
-Download all videos on a playlist:  
-    ```viddl-rb http://www.youtube.com/playlist?list=PL7E8DA0A515924126```
-
-Download all videos from a user:  
-    ```viddl-rb http://www.youtube.com/user/tedtalksdirector```
-
-Filter videos to download from a user/playlist:  
-    ```viddl-rb http://www.youtube.com/user/tedtalksdirector --filter /internet/i```
-
-The --filter argument accepts a regular expression and will only download videos where the title matches the regex.
-It uses the same syntax as Ruby regular expression literals do.
-
-__Library Usage:__
-
-```ruby
-require 'viddl-rb'
-
-download_urls = ViddlRb.get_urls("http://www.youtube.com/watch?v=QH2-TGUlwu4")
-download_urls.first     # => "http://o-o.preferred.arn06s04.v3.lscac ..."
-```
-
-The ViddlRb module has the following module public methods:
-
-* __get_urls_names(url)__
--- Returns an array of one or more hashes that has the keys :url which
-points to the download url and :name which points to the name 
-(which is a filename safe version of the video title with a file extension).
-Returns nil if the url is not recognized by any plugins.
-
-* __get_urls_exts(url)__
--- Same as get_urls_names but with just the file extension (for example ".mp4")
-instead of the full filename, and the :name key is replaced with :ext.
-Returns nil if the url is not recognized by any plugins.
-
-* __get_urls(url)__
--- Returns an array of download urls for the specified video url.
-Returns nil if the url is not recognized by any plugins.
-
-* __get_names(url)__
--- Returns an array of filenames for the specified video url.
-Returns nil if the url is not recognized by any plugins.
-
-* __io=(io_object)__
--- By default all plugin output to stdout will be suppressed when the library is used.
-If you are interested in the output of a plugin, you can set an IO object that
-will receive all plugin output using this method. For example:
-
-```ruby
-require 'viddl-rb'
-
-ViddlRb.io = $stdout     # plugins will now write their output to $stdout
-```
-
-All the __get__ methods in the ViddlRb module will raise either a ViddlRb::PluginError or a ViddlRb::DownloadError if the plugin fails. 
-A ViddlRb::PluginError is raised if the plugin fails in an unexpected way, and a ViddlRb::DownloadError is raised if the video could not be downloaded for some reason. 
-An example of that is if a Youtube video is not embeddable - then it can't be downloaded.
-
-```ruby
-begin
-  ViddlRb.get_urls(video_url)
-rescue ViddlRb::DownloadError => e
-  puts "Could not get download url: #{e.message}"
-rescue ViddlRb::PluginError => e
-  puts "Plugin blew up! #{e.message}\n" +
-       "Backtrace:\n#{e.backtrace.join("\n")}"
-end
-```
-
-__Requirements:__
-
-* curl/wget or the [progress bar](http://github.com/nex3/ruby-progressbar/) gem  
-* [Nokogiri](http://nokogiri.org/)
-* [Mechanize](http://mechanize.rubyforge.org/)
-* ffmpeg if you want to extract audio tracks from the videos
-
-__Co Maintainer:__
-* [kl](https://github.com/kl): Windows support (who knew!), bug fixes, veoh plugin, metacafe plugin, refactoring it into a library, ...
-
-__Contributors:__  
-* [divout](https://github.com/divout) aka Ivan K: blip.tv plugin, bugfixes
-* Sniper: bugfixes
-* [Serabe](https://github.com/Serabe) aka Sergio Arbeo: packaging viddl as a binary
-* [laserlemon](https://github.com/laserlemon): Adding gemnasium images to readme
+__viddl-rb:__  
+Created by Marc Seeger (@rb2k)  
+Repo: http://github.com/rb2k/viddl-rb  
+[![Build Status](https://secure.travis-ci.org/rb2k/viddl-rb.png)](http://travis-ci.org/rb2k/viddl-rb)
+
+
+
+__Installation:__  
+gem install viddl-rb
+
+__Usage:__  
+
+Download a video:  
+    viddl-rb http://www.youtube.com/watch?v=QH2-TGUlwu4
+
+Download a video and extract the audio:  
+    viddl-rb http://www.youtube.com/watch?v=QH2-TGUlwu4 --extract-audio 
+
+In both cases we'll name the output file according to the video title.
+
+
+__Requirements:__
+
+* curl/wget or the [progress bar](http://github.com/nex3/ruby-progressbar/) gem  
+* [Nokogiri](http://nokogiri.org/)
+* ffmpeg if you want to extract audio tracks from the videos
+
+
+__Contributors:__
+
+* [kl](https://github.com/kl): Windows support (who knew!), bug fixes, veoh plugin, metacafe plugin   
+* [divout](https://github.com/divout) aka Ivan K: blip.tv plugin, bugfixes
+* Sniper: bugfixes
+* [Serabe](https://github.com/Serabe) aka Sergio Arbeo: packaging viddl as a binary

data/Rakefile

@@ -1,22 +1,8 @@
-require 'rubygems'
-require 'bundler/setup'
-require 'rake/testtask'
-
-task :default => [:test]
-
-Rake::TestTask.new(:test) do |t|
-  #t.pattern = "spec/*_spec.rb"
-  t.test_files = ["spec/lib_spec.rb", "spec/url_extraction_spec.rb",  "spec/integration_spec.rb"]
-end
-
-Rake::TestTask.new(:test_lib) do |t|
-  t.test_files = FileList["spec/lib_spec.rb"]
-end
-
-Rake::TestTask.new(:test_extract) do |t|
-  t.test_files = FileList["spec/url_extraction_spec.rb"]
-end
-
-Rake::TestTask.new(:test_integration) do |t|
-  t.test_files = FileList["spec/integration_spec.rb"]
-end
+require 'rubygems'
+require 'rake/testtask'
+
+task :default => [:test]
+
+Rake::TestTask.new do |t|
+  t.pattern = "spec/*_spec.rb"
+end

data/bin/viddl-rb

@@ -1,45 +1,117 @@
-#!/usr/bin/env ruby
-$LOAD_PATH << File.join(File.dirname(__FILE__), '..', 'helper')  # general helpers
-$LOAD_PATH << File.join(File.dirname(__FILE__), 'helper')  	     # bin-specific helpers
-
-require "rubygems"
-require "nokogiri"
-require "mechanize"
-require "cgi"
-require "open-uri"
-require "open3"
-require "optparse"
-
-require "driver.rb"
-require "downloader.rb"
-require "download-helper.rb"
-require "parameter-parser.rb"
-require "plugin-helper.rb"
-require "audio-helper.rb"
-require "utility-helper.rb"
-
-begin
-  # params is a hash with keys for each of the parameters passed in.
-  # see helper/parameter-parser.rb for what those keys are.
-  params = ParameterParser.parse_app_parameters(ARGV)
-
-  puts "Loading Plugins"
-  ViddlRb::UtilityHelper.load_plugins
-  puts "Plugins loaded: #{ViddlRb::PluginBase.registered_plugins.inspect}"
-
-  puts "Will try to extract audio: #{params[:extract_audio] == true}."
-  puts "Analyzing URL: #{params[:url]}"
-
-  app = Driver.new(params)   
-  app.start                  # starts the download process
-
-rescue OptionParser::ParseError, ViddlRb::RequirementError => e
-  puts "Error: #{e.message}"
-  exit(1)
-
-rescue StandardError => e
-  puts "Error: #{e.message}"
-  puts "\nBacktrace:"
-  puts e.backtrace
-  exit(1)
-end
+#!/usr/bin/env ruby
+$LOAD_PATH << File.join(File.dirname(__FILE__), '..', 'helper')
+
+require "rubygems"
+require "nokogiri"
+require "cgi"
+require "open-uri"
+require "open3"
+require "download-helper.rb"
+require "plugin-helper.rb"
+
+if ARGV[0].nil?
+	puts "Usage: viddl-rb URL [--extract-audio]"
+	exit
+end
+
+puts "Loading Plugins"
+Dir[File.join(File.dirname(__FILE__),"../plugins/*.rb")].each do |plugin|
+	load plugin
+end
+
+puts "Plugins loaded: #{PluginBase.registered_plugins.inspect}"
+
+url = ARGV[0]
+extract_audio = ARGV.include?('--extract-audio')
+url_only = ARGV.include?('--url-only')
+title_only = ARGV.include?('--title-only')
+
+puts "Will try to extract audio: #{extract_audio}."
+
+unless url.match(/^http/)
+	puts "Please include 'http' with your URL e.g. http://www.youtube.com/watch?v=QH2-TGUlwu4"
+	exit(1)
+end
+
+puts "Analyzing URL: #{url}"
+#Check all plugins for a match
+PluginBase.registered_plugins.each do |plugin|
+	if plugin.matches_provider?(url)
+		puts "#{plugin}: true"
+		begin
+			#we'll end up with an array of hashes with they keys :url and :name 
+			download_queue = plugin.get_urls_and_filenames(url)
+		rescue StandardError => e
+			puts "Error while running the #{plugin.name.inspect} plugin. Maybe it has to be updated? Error: #{e.message}."	
+			exit(1)
+		end
+
+		if url_only
+			download_queue.each{|url_name| puts url_name[:url]}
+			exit
+		elsif title_only
+			download_queue.each{|url_name| puts url_name[:name]}
+			exit
+		end
+
+		download_queue.each do |url_name|
+			result = DownloadHelper.save_file(url_name[:url], url_name[:name])
+			if result
+				puts "Download for #{url_name[:name]} successful."
+				if extract_audio
+					puts "Extracting audio for #{url_name[:name]}"
+					if DownloadHelper.os_has?('ffmpeg')
+						no_ext_filename = url_name[:name].split('.')[0..-1][0]
+						#capture stderr because ffmpeg expects an output param and will error out
+						puts "Gathering information about the downloaded file."
+						file_info = Open3.popen3("ffmpeg -i #{url_name[:name]}") {|stdin, stdout, stderr, wait_thr| stderr.read }
+						puts "Done gathering information about the downloaded file."
+						if !file_info.to_s.empty?
+						  audio_format_matches = file_info.match(/Audio: (\w*)/)
+						  if audio_format_matches
+						  	audio_format = audio_format_matches[1]
+						  	puts "detected audio format: #{audio_format}"
+						  else
+						  	puts "Couldn't find any audio:\n#{file_info.inspect}"
+						  	next
+						  end
+						  
+						  extension_mapper = {
+							'aac' => 'm4a',
+							'mp3' => 'mp3',
+							'vorbis' => 'ogg'
+						  }
+
+						  if extension_mapper.key?(audio_format)
+						  	output_extension = extension_mapper[audio_format]
+						  else
+							#lame fallback
+							puts "Unknown audio format: #{audio_format}, using name as extension: '.#{audio_format}'."
+							output_extension = audio_format
+						  end
+						  output_filename = "#{no_ext_filename}.#{output_extension}"
+						  if File.exist?(output_filename)
+						  	puts "Audio file seems to exist already, removing it before extraction."
+						    File.delete(output_filename)
+						  end
+						  Open3.popen3("ffmpeg -i #{url_name[:name]} -vn -acodec copy #{output_filename}") {|stdin, stdout, stderr, wait_thr| stdout.read }
+						  puts "Done extracting audio to #{output_filename}"
+						else
+						  puts "Error while checking audio track of #{url_name[:name]}"
+						end
+					else
+						puts "Didn't detect ffmpeg on your system, can't extract audio."
+					end
+				end
+			else
+				puts "Download for #{url_name[:name]} failed."
+			end
+		end
+		#plugin matched and downloaded, we're done
+		exit
+	else
+		puts "#{plugin}: false"
+	end
+end
+
+puts "No plugin seems to feel responsible for this URL."

data/helper/download-helper.rb

@@ -1,94 +1,67 @@
-module ViddlRb
-
-  class RequirementError < StandardError; end
-
-  class DownloadHelper
-
-    #viddl will use the first of these tools it finds on the system to download the video.
-    #if the system does not have any of these tools, net/http is used instead.
-    TOOLS_PRIORITY_LIST = [:wget, :curl]
-
-    #simple helper that will save a file from the web and save it with a progress bar
-    def self.save_file(file_url, file_name, opts = {})
-      trap("SIGINT") { puts "goodbye"; exit }
-
-      #default options
-      options = {:save_dir => ".",
-                 :amount_of_retries => 6,
-                 :tool => get_tool}
-
-      opts[:tool] = options[:tool] if opts[:tool].nil?
-      options.merge!(opts)
-
-      file_path = File.expand_path(File.join(options[:save_dir], file_name))
-      success = false
-
-      #Some providers seem to flake out every now end then
-      options[:amount_of_retries].times do |i|
-        case options[:tool].to_sym
-        when :wget
-          puts "Using wget"
-          success = system "wget \"#{file_url}\" -O #{file_path.inspect}"
-        when :curl
-          puts "Using curl"
-          #-L means: follow redirects, We set an agent because Vimeo seems to want one
-          success = system "curl -A 'Wget/1.8.1' --retry 10 --retry-delay 5 --retry-max-time 4  -L \"#{file_url}\" -o #{file_path.inspect}"
-        else
-          require_progressbar
-          puts "Using net/http"
-          success = download_and_save_file(file_url, file_path)
-        end
-        #we were successful, we're outta here
-        if success
-          break
-        else
-          puts "Download seems to have failed (retrying, attempt #{i+1}/#{options[:amount_of_retries]})"
-          sleep 2
-        end
-      end
-      success
-    end
-
-    def self.get_tool
-      tool = TOOLS_PRIORITY_LIST.find { |tool| ViddlRb::UtilityHelper.os_has?(tool) }
-      tool || :net_http
-    end
-
-    def self.require_progressbar
-      begin
-        require "progressbar"
-      rescue LoadError
-        raise RequirementError,
-          "you don't seem to have curl or wget on your system. In this case you'll need to install the 'progressbar' gem."
-      end
-    end
-
-    # downloads and saves a file using the net/http streaming api
-    # return true if the download was successful, else returns false
-    def self.download_and_save_file(download_url, full_path)
-      final_url = UtilityHelper.get_final_location(download_url)    # follow all redirects
-      uri = URI(final_url)
-      file = File.new(full_path, "wb")
-      file_size = 0
-
-      Net::HTTP.start(uri.host, uri.port) do |http|
-        http.request_get(uri.request_uri) do |res|
-          file_size = res.read_header["content-length"].to_i
-          bar = ProgressBar.new(File.basename(full_path), file_size)
-          bar.file_transfer_mode
-          res.read_body do |segment|
-            bar.inc(segment.size)
-            file.write(segment)
-          end
-        end
-      end
-      file.close
-      print "\n"
-      download_successful?(full_path, file_size)   #because Net::HTTP.start does not throw Net exceptions
-    end
-
-    def self.download_successful?(full_file_path, file_size)
-      File.exist?(full_file_path) && File.size(full_file_path) == file_size
-    end
-  end
-end
+class DownloadHelper
+#usually not called directly
+  def self.fetch_file(uri)
+
+  begin
+  	require "progressbar" #http://github.com/nex3/ruby-progressbar
+  rescue LoadError
+  	puts "ERROR: You don't seem to have curl or wget on your system. In this case you'll need to install the 'progressbar' gem."
+  	exit
+  end
+    progress_bar = nil 
+    open(uri, :proxy => nil,
+      :content_length_proc => lambda { |length|
+        if length && 0 < length
+          progress_bar = ProgressBar.new(uri.to_s, length)
+        end 
+      },
+      :progress_proc => lambda { |progress|
+        progress_bar.set(progress) if progress_bar
+      }) {|file| return file.read}        
+  end
+  
+  #simple helper that will save a file from the web and save it with a progress bar
+  def self.save_file(file_uri, file_name)
+    unescaped_uri = CGI::unescape(file_uri)
+    result = false
+    if os_has?("wget")
+      puts "using wget"
+      result = system("wget \"#{unescaped_uri}\" -O #{file_name}")
+    elsif os_has?("curl")
+      puts "using curl"
+      #-L means: follow redirects, We set an agent because Vimeo seems to want one
+    	result = system("curl -A 'Wget/1.8.1' -L \"#{unescaped_uri}\" -o #{file_name}")
+    else
+    	puts "using net/http"
+      open(file_name, 'wb') { |file|   	      
+        file.write(fetch_file(unescaped_uri)); puts
+      }
+      result = true
+    end         
+    result
+  end
+  
+  #checks to see whether the os has a certain utility like wget or curl
+  def self.os_has?(utility)
+    windows = ENV['OS'] =~ /windows/i
+    return `which #{utility}`.include?(utility) unless windows # if not Windows
+
+    #use where (simliar to which) if present to reduce console clutter
+    begin
+      has_where? ? `where #{utility}` : `#{utility}`
+      return true
+    rescue Errno::ENOENT
+      return false
+    end
+  end
+
+  #checks if Windows has the where utility (Server 2003 and later)
+  def self.has_where?
+    begin
+      `where`
+      true
+    rescue Errno::ENOENT
+      false
+    end
+  end
+end