torrents 1.0.0

data/.gitignore

@@ -0,0 +1,5 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*
+authentication/*

data/.rspec

@@ -0,0 +1,5 @@
+--color
+-fs
+-Ilib
+-Ispec
+--require spec_helper

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in torrents.gemspec
+gemspec

data/README.md

@@ -0,0 +1,188 @@
+# Torrents
+
+Search and download torrents from your favorite bittorrent tracker using Ruby.
+
+Download and get information like:
+
+- Subtitles, english and swedish.
+- Movie information (if the torrent is a movie), actors, grade, original title, length, trailers and so on.
+- A direct download link to the torrent.
+- [IMDB](http://imdb.com) link
+
+## Which trackers are implemented at the moment?
+
+### Open trackers
+
+- [The Pirate Bay](http://thepiratebay.org/)
+
+### Closed trackers
+
+- [TTI](http://tti.nu/)
+- [Torrentleech](http://www.torrentleech.org/)
+
+## How to use
+
+### Search for torrents
+
+    >> Torrents.the_pirate_bay.search("chuck").results
+    
+### List recent torrents
+
+    >> Torrents.the_pirate_bay.results
+    
+### List recent torrents - with category
+
+    >> Torrents.the_pirate_bay.category(:movies).results
+    
+### Specify a page
+
+The `page` method can be places anywhere before the `results` method.
+
+It starts counting from `1` and goes up, no matter what is used on the site it self.
+
+    >> Torrents.the_pirate_bay.page(6).results
+
+### Specify some cookies
+
+Some trackers requires cookies to work, even though [The Pirate Bay](http://thepiratebay.org/) is not one of them.
+
+    >> Torrents.the_pirate_bay.cookies(user_id: "123", hash: "c4656002ce46f9b418ce72daccfa5424").results
+
+## What methods to work with
+
+### The results method
+
+As soon as you apply the `results` method on the query it will try to execute your request.
+If you for example want to activate the debugger, define some cookies or specify a page, then you might do something like this.
+      
+      $ Torrents.the_pirate_bay.page(5).debug(true).cookies(:my_cookie => "value").results
+      
+It will return a list of `Container::Torrent` object if the request was sucessfull, otherwise an empty list.
+
+### The find_by_details method
+
+If you have access to a single details link and want to get some useful data from it, then `find_by_details` might fit you needs.
+
+The method takes the url as an argument and returns a single `Container::Torrent` object.
+
+    $ Torrents.the_pirate_bay.find_by_details("http://thepiratebay.org/torrent/6173093/")
+
+# What data to work with
+
+### The Container::Torrent class
+
+The class has some nice accessors that might be useful.
+
+- **title** (String) The title.
+- **details** (String) The url to the details page.
+- **seeders** (Fixnum) The amount of seeders.
+- **dead?** (Boolean) Check to see if the torrent has no seeders. If it has no seeders, then `dead?` will be true.
+- **torrent** (String) The url. This should be a direct link to the torrent.
+- **id** (Fixnum) An unique id for the torrent. The id is only unique for this specific torrent, not all torrents.
+- **tid** (String) The `tid` method, also known as `torrent id` is a *truly* unique identifier for all torrents. It is generated using a [MD5](http://sv.wikipedia.org/wiki/MD5) with the torrent domain and the `id` method as a seed.
+- **torrent_id** (String) The same as the `tid` method.
+- **imdb** (String) The imdb link for the torrent, if the details view contains one. 
+- **imdb_id** (String) The imdb id for the torrent, if the details view contain one. Example: tt0066026.
+- **subtitle** ([Undertexter](https://github.com/oleander/Undertexter)) The subtitle for the torrent. Takes one argument, the language for the subtitle. Default is `:english`. Read more about it [here](https://github.com/oleander/Undertexter).
+- **movie** ([MovieSearcher](https://github.com/oleander/MovieSearcher)) Read more about the returned object at the [MovieSearcher](https://github.com/oleander/MovieSearcher) project page.
+
+**Note:** The `seeders`, `movie`, `subtitle`, `imdb_id` and `ìmdb` method will do another request to the tracker, which means that it will take a bit longer to load then the other methods.
+
+## What cookies to pass
+
+Here is an example
+
+    $ Torrents.torrentleech.cookies({:member_id => "123", :pass_hash => "value", :PHPSESSID => "value"}).results 
+    
+All values you pass to `cookies` must be of type string, like in the example above.
+
+- Torrentleech
+  - member_id
+  - pass_hash
+  - PHPSESSID
+- TTI
+  - hass
+  - pass
+  - uid
+
+**Note:** The cookies you pass might be browser and IP-adress sensitive. Which means that it might only work in the current browser using the current Internet connection.
+  
+## Error handling
+
+I decided in the beginning of the project to rescue parse errors during the runtime and instead print them as warnings.
+
+### Why? - Lack of good selectors
+
+The trackers parser, [this](https://github.com/oleander/Torrents/blob/master/lib/torrents/trackers/the_pirate_bay.rb) one for example, isn't always returning the right data. 
+
+Due to the lack of useful CSS selectors on the given tracker. It returns 32 rows, the first and the last containing the header and the footer of the table.
+The unwanted results will be thrown away by the [validator](https://github.com/oleander/Torrents/blob/master/lib/torrents/container.rb#L141), but may raise errors during the run time.
+The easiest way to solve it was to just isolate the tracker, if it raised an error we return nil.
+
+### Get the error messages
+
+You can read errors in two ways.
+
+Activate the debugger by adding the `debug` method to your query. The errors will be printed as warnings in the console.
+  
+    $ Torrents.the_pirate_bay.debug(true).results
+
+Request a list of errors using the `errors` method.
+
+    $ Torrents.the_pirate_bay.errors
+    >> ["...undefined method `attr' for nil:NilClass>...", "32 torrents where found, 2 where not valid", "..."]
+
+## How do access tracker X
+
+Here is how to access an implemented tracker.
+The first static method to apply is the name of the tracker in lower non camel cased letters.
+
+The Pirate Bay becomes `the_pirate_bay`, TTI becomes `tti` and Torrentleech `torrentleech`.
+
+Here is an example.
+
+    $ Torrents.torrentleech.cookies({:my_cookie => "value"}).results 
+
+Take a look at the [tests](https://github.com/oleander/Torrents/tree/master/spec/trackers) for all trackers to get to know more.
+
+## Add you own tracker
+
+I'm about to write a wiki that describes how to add you own site.
+Until then, take a look at the parser for [The Pirate Bay](https://github.com/oleander/Torrents/blob/master/lib/torrents/trackers/the_pirate_bay.rb).
+
+All heavy lifting has already been done, so adding another tracker should be quite easy.
+
+I'm using [Nokogiri](http://nokogiri.org/) to parse data from the site, which in most cases means that you don't have to mess with regular expressions.
+
+Don't know Nokogiri? Take a look at [this](http://railscasts.com/episodes/190-screen-scraping-with-nokogiri) awesome screen cast by [Ryan Bates](https://github.com/ryanb). 
+    
+### The short version
+
+1. Create your own fork of the project.
+2. Create and implement a tracker file inside the [tracker directory](https://github.com/oleander/Torrents/tree/master/lib/torrents/trackers).
+3. Add a cached version of the tracker [here](https://github.com/oleander/Torrents/tree/master/spec/data). **Note:** Remember to remove sensitive data from the cache like user name and uid.
+4. Add tests for it, [here](https://github.com/oleander/Torrents/blob/master/spec/trackers/the_pirate_bay_spec.rb) is a skeleton for the Pirate Bay class to use as a start.
+5. Add the tracker to the readme.
+6. Do a pull request, if you want to share you implementation with the world.
+
+You don't have to take care about exceptions, `Torrents` does that for you.
+
+## Disclaimer
+
+Before you use `Torrents` make sure you have permission from the tracker in question to use their data.
+
+## How do install
+
+    [sudo] gem install torrents
+    
+## How to use it in a rails 3 project
+
+Add `gem 'torrents'` to your Gemfile and run `bundle`.
+
+## Requirements
+
+Torrents is tested in OS X 10.6.6 using Ruby 1.9.2.
+
+## License
+
+Torrents is released under the MIT license.

data/Rakefile

@@ -0,0 +1,2 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks

data/lib/torrents.rb

@@ -0,0 +1,140 @@
+$:.push File.expand_path("../../lib/torrents", __FILE__)
+$:.push File.expand_path("../../lib/torrents/trackers", __FILE__)
+
+require 'rest_client'
+require 'nokogiri'
+require 'torrents/container'
+
+class Torrents < Container::Shared
+  attr_accessor :page
+  
+  def initialize
+    @torrents = []
+    @errors   = []
+    @url = {
+      callback: lambda { |obj|
+        obj.send(:inner_recent_url)
+      },
+      search: {
+        value: ""
+      }
+    }
+  end
+  
+  def exists?(tracker)
+    File.exists?(File.dirname(File.expand_path( __FILE__)) + "/torrents/trackers/" + tracker.to_s + ".rb")
+  end
+  
+  def content
+    @content ||= Nokogiri::HTML(self.download(self.url))
+  end
+  
+  # Set the default page
+  def inner_page
+    ((@page ||= 1) - 1 + self.inner_start_page_index).to_s
+  end
+
+  def url
+    @url[:callback].call(self).
+      gsub('<SEARCH>', @url[:search][:value]).
+      gsub('<PAGE>', self.inner_page)
+  end
+  
+  # Makes this the {tracker} tracker
+  def add(tracker)
+    @tracker = tracker.to_s; self
+  end
+  
+  def page(value)
+    @page = value
+    raise ArgumentError.new("To low page value, remember that the first page has the value 1") if self.inner_page.to_i < 0
+    self
+  end
+  
+  def debugger(value)
+    @debug = value; self
+  end
+  
+  # Set the search value
+  def search(value)
+    @url.merge!(:callback => lambda { |obj|
+      obj.send(:inner_search_url)
+    }, :search => {:value => value}); self
+  end
+  
+  def category(cat)
+    @url.merge!(:callback => lambda { |obj|
+      obj.send(:inner_category_url, cat)
+    }); self
+  end
+  
+  def cookies(args)
+    @cookies = args; self
+  end
+  
+  # If the user is trying to do some funky stuff to the data
+  def method_missing(method, *args, &block)
+    return self.inner_call($1.to_sym, args.first) if method =~ /^inner_(.+)$/
+    super(method, args, block)
+  end
+  
+  def self.method_missing(method, *args, &block)
+    this = Torrents.new
+    # Raises an exception if the site isn't in the trackers.yaml file
+    raise Exception.new("The site #{method} does not exist") unless this.exists?(method)
+    
+    # Yes, I like return :)
+    return this.add(method)
+  end
+  
+  # Returns a Container::Torrent object
+  # {details} (String) The details url for the torrent
+  def find_by_details(details)
+    self.create_torrent({
+      details: details,
+      tracker: @tracker
+    })
+  end
+  
+  # Creates a torrent based on the ingoing arguments
+  # Is used by {find_by_details} and the {results} method
+  # Returns a Container::Torrent object
+  # {arguments} (Hash) The params to the Torrent constructor
+  # The debugger and cookie param is passed by default
+  def create_torrent(arguments)
+    arguments.merge!(:debug => @debug) if @debug
+    arguments.merge!(:cookies => @cookies) if @cookies
+    Container::Torrent.new(arguments)
+  end
+  
+  # Returns errors from the application.
+  # Return type: A list of strings
+  def errors
+    self.results; @errors.uniq
+  end
+  
+  def results
+    return @torrents if @torrents.any?
+    counter  = 0
+    rejected = 0
+    self.inner_torrents(self.content).each do |tr|
+      counter += 1
+      
+      torrent = self.create_torrent({
+        details: self.inner_details(tr),
+        torrent: self.inner_torrent(tr),
+        title: self.inner_title(tr).to_s.strip,
+        tracker: @tracker
+      })
+      
+      if torrent.valid?
+        @torrents << torrent
+      else
+        rejected += 1
+      end 
+    end
+    
+    @errors << "#{counter} torrents where found, #{rejected} where not valid" unless rejected.zero?
+    return @torrents
+  end
+end

data/lib/torrents/container.rb

@@ -0,0 +1,246 @@
+module Container
+  require "rest_client"
+  require "nokogiri"
+  require 'rchardet19'
+  require "iconv"
+  require "classify"
+  require "digest/md5"
+  require "movie_searcher"
+  require "undertexter"
+  
+  # Loads all trackers inside the trackers directory
+  Dir["#{File.dirname(File.expand_path(__FILE__))}/trackers/*.rb"].each {|rb| require "#{rb}"}
+
+  class Shared
+    include Trackers
+    # Downloads the URL, returns an empty string if an error occurred
+    # Here we try to convert the downloaded content to UTF8, 
+    # if we"re at least 60% sure that the content that was downloaded actally is was we think
+    # The timeout is set to 10 seconds, after that time, an empty string will be returned 
+    # {url} (String) The URL to download
+    def download(url)
+      begin
+        data = RestClient.get self.url_cleaner(url), {:timeout => 10, :cookies => @cookies}
+        cd = CharDet.detect(data)
+        return (cd["confidence"] > 0.6) ? (Iconv.conv(cd["encoding"] + "//IGNORE", "UTF-8", data) rescue data) : data
+      rescue
+        self.error("Something when wrong when trying to fetch #{url}", $!)
+      end
+      
+      # The default value, if {RestClient} for some reason craches (like wrong encoding or a timeout)
+      return ""
+    end
+    
+    # Prints a nice(er) error to the console if something went wrong
+    # This is only being called when trying to download or when trying to parse a page
+    # {messages} (String) The custom error to the user
+    # {error} (Exception) The actual error that was thrown
+    # TODO: Implement a real logger => http://www.ruby-doc.org/stdlib/libdoc/logger/rdoc/classes/Logger.html
+    def error(messages, error = "")
+      messages = messages.class == Array ? messages : [messages]
+      error = error.inspect[0..60]
+      @errors = [] unless @errors
+      @errors << "#{messages.join(", ")}\n#{error}"
+      
+      return unless @debug
+      
+      warn "An error in the Torrents gem occurred"
+      warn "==> " + messages.join("\n\t")
+      warn "==> " + error + " ..."
+      warn "\n\n"
+    end
+    
+    # A middle caller that can handle errors for external trackers
+    # If the tracker that is being loaded in {load} crashes, 
+    # then this method makes sure that the entire application won"t crash
+    # {method} (Hash) The method that is being called inside the trackers module
+    # {tr} (Nokogiri | Symbol) The object that contains the HTML content of the current row
+    def inner_call(method, option = nil)
+      begin
+        results = option.nil? ? self.load.send(method) : self.load.send(method, option) if self.valid_option?(method, option)
+      rescue
+        self.error("An error occurred in the #{@tracker} class at the #{method} method.", $!)
+      ensure
+        raise NotImplementedError.new("#{option} is not implemented yet") if results.nil? and method == :category_url
+        value = results.nil? ? self.default_values(method) : results
+      end
+      
+      return value
+    end
+    
+    # Returns default value if any of the below methods (:details for example) return an exception.
+    # If the method for some reason isn't implemented (is not in the hash below), then it will return an empty string
+    # {method} (Hash) The method that raised an exception 
+    def default_values(method)
+      # warn "Something went wrong, we can't find the #{method} tag, using default values"
+      {torrent: "", torrents: [], seeders: 1, title: "", details: "", id: 0}[method] || ""
+    end
+    
+    # Creating a singleton of the {tracker} class
+    def load
+      @load ||= eval("#{Classify.new.camelize(@tracker)}.new")
+    end
+    
+    # Check to see if the ingoing arguments to the tracker if valid.
+    # If something goes wrong after the parser has been implemented, then it (the tracker) wont crash.
+    # Insted we write to a log file, so that the user can figure out the problem afterwards.
+    # {method} (Symbol) That method that is being called
+    # {option} (Object) That params to the {method}, can be anything, including {nil}
+    # Returns a boolean, {true} if the {method} can handle the {option} params, {false} otherwise.
+    def valid_option?(method, option)
+      case method
+        when :details, :title, :torrent
+          option.instance_of?(Nokogiri::XML::Element)
+        when :category_url
+          option.instance_of?(Symbol)
+        when :torrents, :seeders
+          option.instance_of?(Nokogiri::HTML::Document)
+        when :id
+          option.instance_of?(String)
+        else
+          true
+      end
+    end
+    # Cleans up the URL
+    # The ingoing param to the {open | RestClient} method can handle the special characters below.
+    # The only way to download the content that the URL points to is to escape those characters.
+    # Read more about it here => http://stackoverflow.com/questions/4999322/escape-and-download-url-using-ruby
+    # {url} (String) The url to escape
+    # Returns an escaped string
+    def url_cleaner(url)
+      url.gsub(/\{|\}|\||\\|\^|\[|\]|\`|\s+/) { |m| CGI::escape(m) }
+    end
+  end
+  
+  class Torrent < Shared
+    attr_accessor :details
+    
+    def initialize(args)
+      args.keys.each { |name| instance_variable_set "@" + name.to_s, args[name] }
+      @errors = [] unless @errors
+      @debug = false unless @debug
+    end
+    
+    # Is the torrent dead?
+    # The definition of dead is; no seeders
+    # Returns a boolean
+    def dead?
+      self.seeders <= 0
+    end
+    
+    # Returns the amount of seeders for the current torrent
+    # If the seeder-tag isn't found, the value one (1) will be returned.
+    # Returns an integer from 0 to inf
+    def seeders
+      @seeders ||= self.inner_call(:seeders, self.content).to_i
+    end
+    
+    # Is the torrent valid?
+    # The definition of valid:
+    #   Non of the accessors
+    #   => is nil
+    #   => contains htmltags
+    #   => starts or ends with whitespace
+    # It must also stand up to the following requirements
+    #   => The details and torrent url must be valid
+    #   => The id for the torrent must only contain integers.
+    # Returns {true} or {false}
+    def valid?
+      [:details, :torrent, :title, :id].each do |method|
+        data = self.send(method)
+        return false if self.send(method).nil? or 
+          data.to_s.empty? or 
+          data.to_s.match(/<\/?[^>]*>/) or 
+          data.to_s.strip != data.to_s
+      end
+      
+      return [
+        !! self.valid_url?(self.details),
+        !! self.valid_torrent?(self.torrent),
+        !! self.inner_call(:id, self.details).to_s.match(/^\d+$/)
+      ].all?
+    end
+    
+    # Downloads the detailed view for this torrent
+    # Returns an Nokogiri object
+    def content
+      @content ||= Nokogiri::HTML self.download(@details)
+    end
+    
+    # Check to see if the ingoing param is a valid url or not
+    def valid_url?(url)
+      !! url.match(/(ftp|http|https):\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-\/]))?/i)
+    end
+    
+    # Check to see if the ingoing param is a valid torrent url or not
+    # The url has to be a valid url and has to end with .torrent
+    def valid_torrent?(torrent)
+      torrent.match(/\.torrent$/) and self.valid_url?(torrent)
+    end
+    
+    # Generates an id using the details url
+    def id
+      @id ||= self.inner_call(:id, self.details).to_i
+    end
+    
+    # Returns the domain for the torrent, without http or www
+    # If the domain for some reason isn't found, it will use an empty string
+    def domain
+      @domain ||= self.details.match(/(ftp|http|https):\/\/([w]+\.)?(.+?\.[a-z]{2,3})/i).to_a[3] || ""
+    end
+    
+    # Returns a unique id for the torrent based on the domain and the id of the torrent
+    def tid
+      @tid ||= Digest::MD5.hexdigest("#{domain}#{id}")
+    end
+    
+    # Just a mirror method for {tid}, just in case someone don't like the method name tid
+    def torrent_id
+      @torrent_id ||= self.tid
+    end
+    
+    # Returns the full url to the related imdb page
+    # The link is parsed from the details view
+    # Example: http://www.imdb.com/title/tt0066026
+    # Return type: String or nil
+    def imdb
+      @imdb ||= self.content.to_s.match(/((http:\/\/)?([w]{3}\.)?imdb.com\/title\/tt\d+)/i).to_a[1]
+    end
+    
+    # Returns the imdb id for the torrent, including the tt at the beginning
+    # Example: tt0066026
+    # Return type: String or nil
+    def imdb_id
+      @imdb_id ||= self.imdb.to_s.match(/(tt\d+)/).to_a[1]
+    end
+    
+    # Returns an movie_searcher object based on the imdb_id, if it exists, otherwise the torrent title
+    # Read more about it here: https://github.com/oleander/MovieSearcher
+    # Return type: A MovieSearcher object or nil
+    def movie
+      self.imdb_id.nil? ? MovieSearcher.find_by_release_name(self.title, :options => {:details => true}) : MovieSearcher.find_movie_by_id(self.imdb_id)
+    end
+    
+    # Returns the title for the torrent
+    # If the title has't been set from the Torrents class, we will download the details page try to find it there.
+    # Return type: String or nil
+    def title
+      @title ||= self.inner_call(:details_title, self.content)
+      @title = @title.strip unless @title.nil?
+    end
+    
+    # Returns a Undertexter object, if we found a imdb_id, otherwise nil
+    # Read more about it here: https://github.com/oleander/Undertexter
+    # Return type: A single Undertexter object or nil
+    def subtitle(option = :english)
+      @subtitle ||= Undertexter.find(self.imdb_id, language: option).based_on(self.title)
+    end
+    
+    # Returns the torrent for the torrent
+    # If the torrent has't been set from the Torrents class, we will download the details page try to find it there.
+    # Return type: String or nil
+    def torrent
+      @torrent ||= self.inner_call(:details_torrent, self.content)
+    end
+  end
+end