super_crawler 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4f566869c9f06df215b047291bbc08a83e25a176
+  data.tar.gz: ee35c2dcee2dae0289e70a8fb0195c63859bf793
+SHA512:
+  metadata.gz: e7f5c9db1479c83af91879d41f9ea6480142ddf11fba9fcc3bc944f46de90d8b821929582526f245cd6752a4506019f8b2489d08175e5c6fe30faea620c1ea61
+  data.tar.gz: 1c81fc244e6679cfe6782b9651782944306290a9281277658a220155ef6c4a7ccee59b9b7a2f058c0862d52027e7f8453ba01a21c953d2f31ff5980e9fea87ad

data/.gitignore

@@ -0,0 +1,11 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+
+*.gem

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.3.1
+before_install: gem install bundler -v 1.12.5

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,49 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at htaidirt@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in super_crawler.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Hassen Taidirt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,255 @@
+# SuperCrawler
+
+Easy (yet efficient) ruby gem to crawl your favorite website.
+
+## Quick Start
+
+Open your terminal, then:
+
+```bash
+$ git clone https://github.com/htaidirt/super_crawler
+
+$ cd super_crawler
+
+$ bundle
+
+$ ./bin/console
+```
+
+```ruby
+ > sc = SuperCrawler::CrawlSite.new('https://gocardless.com')
+
+ > sc.start # => Start crawling the website
+
+ > sc.render(5) # => Show first 5 results of the crawling as sitemap
+```
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'super_crawler'
+```
+
+And then execute:
+
+```ruby
+bundle install
+```
+
+Or install it yourself as:
+
+```ruby
+gem install super_crawler
+```
+
+Want to experiment with the gem without installing it? Clone the following repo and run `bin/console` for an interactive prompt that will allow you to experiment.
+
+## Warning!
+
+This gem is an experiment and can't be used for production purposes. Please, use it with caution if you want to use it in your projects.
+
+There are also a lot of limitations that weren't handled due to time. You'll find more information on the limitations below.
+
+SuperCrawler gem was only tested on MRI and ruby 2.3.1.
+
+## Philosophy
+
+Starting from a URL, extract all the internal links and assets within the page. Add all unique links to an array for future exploration of theses links. Repeat for each link in the links list until no new link is discovered.
+
+Due to the heavy operations, and the time to access each page content, we will use threads to perform near-parallel processing.
+
+In order to keep the code readable and structured, create two classes:
+
+- `SuperCrawler::CrawlPage` that is responsible for crawling a single page and extracting all relevant information (internal links and assets)
+- `SuperCrawler::CrawlSite` that is responsible for crawling a whole website, by collecting links and calling `SuperCrawler::CrawlPage` within threads. This class is also responsible for rendering results.
+
+## More detailed use
+
+Open your favorite ruby console and require the gem:
+
+```ruby
+require 'super_crawler'
+```
+
+### Crawling a single web page
+
+Read the following if you would like to crawl a single web page and extract relevant information (internal links and assets).
+
+```ruby
+page = SuperCrawler::CrawlPage.new( url )
+```
+
+Where `url` should be the URL of the page you would like to crawl.
+
+**Nota:** When missing a scheme (`http://` or `https://`), SuperCrawler will prepend the URL with an `http://`.
+
+#### Get the encoded URL
+
+Run
+
+```ruby
+page.url
+```
+
+to get the encoded URL provided.
+
+#### Get internal links of a page
+
+Run
+
+```ruby
+page.get_links
+```
+
+to get a list of internal links within the crawled page. An internal link is a link that _has the same host than the page URL_. Subdomains are rejected.
+
+This method searches in the `href` attribute of all `<a>` anchor tags.
+
+**Nota:** This method returns an array of absolute URLs (all internal links).
+
+**Nota 2:** Bad links and special links (like mailto and javascript) are discarded.
+
+#### Get images of a page
+
+Run
+
+```ruby
+page.get_images
+```
+
+to get a list of images links within the page. The images links are extracted from the `src="..."` attribute of all `<img>` tags.
+
+**Nota:** Images included using CSS or JavaScript aren't detected by the method.
+
+**Nota 2:** This method returns an array of absolute URLs.
+
+#### Get stylesheets of a page
+
+Run
+
+```ruby
+page.get_stylesheets
+```
+
+to get a list of stylesheets links within the page. The stylesheets links are extracted from the `href="..."` attribute of all `<link rel="stylesheet">` tags.
+
+**Nota:** Inline styling isn't yet detected by the method.
+
+**Nota 2:** This method returns an array of absolute URLs.
+
+#### Get scripts of a page
+
+Run
+
+```ruby
+page.get_scripts
+```
+
+to get a list of scripts links within the page. The scripts links are extracted from the `src="..."` attribute of all `<script>` tags.
+
+**Nota:** Inline script isn't yet detected by the method.
+
+**Nota 2:** This method returns an array of absolute URLs.
+
+#### List all assets of a page
+
+Run
+
+```ruby
+page.get_assets
+```
+
+to get a list of all assets (images, stylesheets and scripts links) as a hash of arrays.
+
+### Crawling a whole web site
+
+First instantiate the site crawler.
+
+```ruby
+sc = SuperCrawler::CrawlSite.new(url, count_threads)
+```
+
+where `url` is the URL of the page to crawl, and `count_threads` the number of threads to handle the job (by default 10).
+
+Next, start the crawler:
+
+```ruby
+sc.start
+```
+
+This can take some time, depending on the site to crawl.
+
+To access crawl results, you can use the following:
+
+```ruby
+sc.links # The array of internal links
+
+sc.crawl_results # Array of hashes containing links and assets for every link crawled
+```
+
+To see the crawling as a sitemap, use:
+
+```ruby
+sc.render(5) # Will render the sitemap of the first 5 pages
+```
+
+TODO: Make more sophisticated rendering class, that can render within files of different formats (HTML, XML, JSON,...)
+
+#### Tips on searching assets and links
+
+After `sc.start`, you can access all collected resources (links and assets) using `sc.crawl_results`. This has the following structure:
+
+```json
+[
+  {
+    url: 'http://example.com/',
+    links: [...array of internal links...],
+    assets: {
+      images: [...array of images links],
+      stylesheets: [...array of stylesheets links],
+      scripts: [...array of scripts links],
+    }
+  },
+  ...
+]
+```
+
+You can use `sc.crawl_results.select{ |resource| ... }` to select a particular resource.
+
+## Limitations
+
+Actually, the gem has the following limitations:
+
+- Subdomains are not considered as internal links
+- Both HTTP and HTTPS pages are taken into account. This can increase the number of links found, but we think that we need to keep it because some sites don't duplicate all contents for HTTP and HTTPS
+- Only links within `<a href="...">` tags are extracted
+- Only images links within `<img src="..."/>` tags are extracted
+- Only stylesheets links within `<link rel="stylesheet" href="..." />` tags are extracted
+- Only scripts links within `<script src="...">` tags are extracted
+- A page that is not accessible (eg. error 404) is not checked later
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/htaidirt/super_crawler. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+Want to contribute, please follow this process:
+
+1. Fork it
+2. Create your feature branch (git checkout -b my-new-feature)
+3. Commit your changes (git commit -am 'Add some feature')
+4. Push to the branch (git push origin my-new-feature)
+5. Create new Pull Request
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+
+## Don't forget to have fun coding Ruby...

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "super_crawler"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/super_crawler/crawl_page.rb

@@ -0,0 +1,160 @@
+require "open-uri"
+require "open_uri_redirections"
+require "nokogiri"
+
+module SuperCrawler
+
+  ###
+  # Crawl a single HTML page
+  # Responsible for extracting all relevant information within a page
+  #
+  class CrawlPage
+
+    attr_reader :url
+
+    def initialize url
+      # Normalize the URL, by adding http(s) if not present in the URL
+      # NOTA: By default, add http:// scheme to an URL that doesn't have one
+      @url = URI.encode( !!(url =~ /^(http(s)?:\/\/)/) ? url : ('http://' + url) )
+    end
+
+    ###
+    # Get INTERNAL links of the page (same host)
+    #
+    def get_links
+      return [] unless page_exists?
+
+      # Get all the links that are within <a> tag, using Nokogiri
+      links = get_doc.css('a').map{ |link| link['href'] }.compact
+
+      # Select only internal links (relative links, or absolute links with the same host)
+      links.select!{ |link| URI.parse(URI.encode link).host.nil? || URI.parse(URI.encode link).host == URI.parse(@url).host }
+
+      # Reject bad matches links (like mailto, tel and javascript)
+      links.reject!{ |link| !!(link =~ /^(mailto:|tel:|javascript:)/) }
+
+      # Clean the links
+      links.map!{ |link| create_absolute_url( link ) } # Make all links absolute
+           .map!{ |link| link.split('#')[0] } # Remove the fragment part from the links (...#...) if any
+           .map!{ |link| URI(URI.encode link).normalize().to_s } # Normalize links
+
+      return links.uniq # Return links without duplicates
+    end
+
+    ###
+    # Get all the images within a page
+    # NOTA: These are images within <img src="..." /> tag.
+    #
+    def get_images
+      return [] unless page_exists?
+
+      # Get all the images sources (URLs), using Nokogiri
+      images_links = get_doc.css('img').map{ |image| image['src'] }.compact
+
+      # Create the absolute path of the images
+      images_links.map!{ |image| create_absolute_url( image ) }
+
+      return images_links.uniq # Return links to images without duplicates
+    end
+
+    ###
+    # Get all the CSS links within a page
+    # NOTA: These are links within <link href="..." /> tag.
+    #
+    def get_stylesheets
+      return [] unless page_exists?
+
+      # Get all the stylesheet links (URLs), using Nokogiri
+      css_links = get_doc.css('link').select{ |css_link| css_link['rel'] == 'stylesheet' }
+                                     .map{ |css_link| css_link['href'] }
+                                     .compact
+
+      # Create the absolute path of the CSS links
+      css_links.map!{ |css_link| create_absolute_url( css_link ) }
+
+      return css_links.uniq # Return links to CSS files without duplicates
+    end
+
+    ###
+    # Get all the JS scripts within a page
+      # NOTA: These are scripts within <script src="..." /> tag.
+    #
+    def get_scripts
+      return [] unless page_exists?
+
+      # Get all the script sources (URLs), using Nokogiri
+      scripts_links = get_doc.css('script').map{ |script| script['src'] }.compact
+
+      # Create the absolute path of the scripts
+      scripts_links.map!{ |script| create_absolute_url( script ) }
+
+      return scripts_links.uniq # Return links to scripts without duplicates
+    end
+
+    ###
+    # Get all assets within a page
+    # Returns a hash of images, stylesheets and scripts URLs
+    #
+    def get_assets
+      {
+        'images': get_images,
+        'stylesheets': get_stylesheets,
+        'scripts': get_scripts
+      }
+    end
+
+    ###
+    # Get links and assets within a page
+    # Returns a hash of links, images, stylesheets and scripts URLs
+    #
+    def get_all
+      {
+        'links': get_links,
+        'images': get_images,
+        'stylesheets': get_stylesheets,
+        'scripts': get_scripts
+      }
+    end
+
+    ###
+    # Check if the page exists
+    #
+    def page_exists?
+      !!( get_doc rescue false )
+    end
+
+    private
+
+    ###
+    # Get the page `doc` (document) from Nokogiri.
+    # Cache it for performace issue.
+    #
+    def get_doc
+      begin
+        @doc ||= Nokogiri(open( @url , allow_redirections: :all ))
+      rescue Exception => e
+        raise "Problem with URL #{@url}: #{e}"
+      end
+    end
+
+    ###
+    # Extract the base URL (scheme and host only)
+    #
+    # eg:
+    # http://mysite.com/abc -> http://mysite.com
+    # https://dev.mysite.co.uk/mylink -> https://dev.mysite.co.uk
+    def base_url
+      "#{URI.parse(@url).scheme}://#{URI.parse(@url).host}"
+    end
+
+    ###
+    # Given a URL, return the absolute URL
+    #
+    def create_absolute_url url
+      # Append the base URL (scheme+host) if the provided URL is relative
+      URI.parse(URI.encode url).host.nil? ? (base_url + url) : url
+    end
+
+  end
+
+end

data/lib/super_crawler/crawl_site.rb

@@ -0,0 +1,154 @@
+require 'thread'
+
+require 'super_crawler/crawl_page'
+
+module SuperCrawler
+
+  ###
+  # Crawl a whole website
+  #
+  class CrawlSite
+
+    attr_reader :links, :crawl_results
+
+    def initialize start_url, threads = 10, options = {}
+      @start_url = URI(URI.encode start_url).normalize().to_s # Normalize the given URL
+      @links = [@start_url] # Will contain the list of all links found
+      @crawl_results = [] # Will contain the crawl results (links and assets), as array of hashes
+      @threads = threads # How many threads to use? Default: 10
+
+      @option_debug = options[:debug].nil? ? true : !!(options[:debug]) # Debug by default
+    end
+
+    ###
+    # Start crawling site
+    # Could take a while. Use threads to speed up crawling and logging to inform user.
+    #
+    def start
+
+      crawling_start_notice # Show message on what will happen
+      threads = [] # Will contain our threads
+      @links_queue = Queue.new # Will contain the links queue that the threads will use
+      @links = [@start_url] # Re-init the links list
+      @crawl_results = [] # Re-init the crawling results
+
+      start_time = Time.now if @option_debug # Start the timer
+
+      # Let's populate our queue with links and resources from source url
+      process_page( @start_url )
+
+      # Create threads to handle new links
+      @threads.times do # Create many threads
+
+        threads << Thread.new do # Add a new threads
+          begin
+            while current_link = @links_queue.pop(true) # Popping every link after another
+              process_page( current_link ) # Get links and assets
+            end
+          rescue ThreadError # Stop when empty links queue
+          end
+        end
+
+      end
+
+      threads.map(&:join) # Activate the threads
+      crawling_summary_notice(start_time, Time.now) if @option_debug # Display crawling summary
+
+      return true
+    end
+
+    ###
+    # Render sitemap
+    # Show, for each link, internal links and assets
+    # We will limit pages to display, because some sites have more than 1,000 pages
+    #
+    def render max_pages = 10
+      draw_line
+      puts "Showing first #{max_links} crawled pages and their contents:\n\n"
+      @crawl_results[0..(max_pages-1)].each_with_index do |result, index|
+        puts "[#{index+1}] Content of #{result[:url]}\n"
+
+        puts "     + Internal links: #{'None' if result[:links].empty?}"
+        result[:links].each { |link| puts "            - #{link}" }
+
+        puts "     + Internal images: #{'None' if result[:assets][:images].empty?}"
+        result[:assets][:images].each { |link| puts "            - #{link}" }
+
+        puts "     + Internal stylesheets: #{'None' if result[:assets][:stylesheets].empty?}"
+        result[:assets][:stylesheets].each { |link| puts "            - #{link}" }
+
+        puts "     + Internal scripts: #{'None' if result[:assets][:scripts].empty?}"
+        result[:assets][:scripts].each { |link| puts "            - #{link}" }
+        puts ""
+      end
+      draw_line
+    end
+
+    private
+
+    ###
+    # Process a page by extracting information and updating links queue, links list and results.
+    #
+    def process_page page_url
+      page = SuperCrawler::CrawlPage.new(page_url) # Crawl the current page
+
+      current_page_links = page.get_links # Get current page internal links
+      new_links = current_page_links - @links # Select new links
+
+      new_links.each { |link| @links_queue.push(link) } # Add new links to the queue
+      @links += new_links # Add new links to the total links list
+      @crawl_results << { # Provide current page crawl result as a hash
+        url: page.url, # The crawled page
+        links: current_page_links, # Its internal links
+        assets: page.get_assets # Its assets
+      }
+
+      log_status( page_url ) if @option_debug # Display site crawling status
+    end
+
+    ###
+    # Display a notice when starting a site crawl
+    #
+    def crawling_start_notice
+      draw_line
+      puts "Start crawling #{@start_url} using #{@threads} threads. Crawling rules:"
+      puts "1. Keep only internal links"
+      puts "2. http and https links are considered different"
+      puts "3. Remove the fragment part from the links (#...)"
+      puts "4. Keep paths with different parameters (?...)"
+      draw_line
+    end
+
+    ###
+    # Log current search status (crawled links / total links)
+    #
+    def log_status url
+      text = "Crawled #{@crawl_results.length.to_s}/#{@links.length.to_s}: #{url}"
+      print "\r#{" "*100}\r" # Clean the previous text
+      print (text.length <= 50) ? text : "#{text[0..46]}..."
+      STDOUT.flush
+    end
+
+    ###
+    # Display final crawling summary after site crawling complete
+    #
+    def crawling_summary_notice time_start, time_end
+      total_time = time_end - time_start
+      puts ""
+      draw_line
+      puts "Crawled #{@links.count} links in #{total_time.to_f.to_s} seconds using #{@threads} threads."
+      puts "Use .crawl_results to access the crawl results as an array of hashes."
+      puts "Use .render to see the crawl_results as a sitemap."
+      draw_line
+    end
+
+    ###
+    # Draw a line (because readability is also important!!)
+    #
+    def draw_line
+      puts "#{'-' * 80}"
+    end
+
+  end
+
+end

data/lib/super_crawler/version.rb

@@ -0,0 +1,3 @@
+module SuperCrawler
+  VERSION = "0.1.0"
+end

data/lib/super_crawler.rb

@@ -0,0 +1,4 @@
+require "super_crawler/version"
+
+require "super_crawler/crawl_page"
+require "super_crawler/crawl_site"

data/super_crawler.gemspec

@@ -0,0 +1,37 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'super_crawler/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "super_crawler"
+  spec.version       = SuperCrawler::VERSION
+  spec.authors       = ["Hassen Taidirt"]
+  spec.email         = ["htaidirt@gmail.com"]
+
+  spec.summary       = %q{Easy (yet efficient) ruby gem to crawl your favorite website.}
+  spec.description   = %q{SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.}
+  spec.homepage      = "https://github.com/htaidirt/super_crawler"
+  spec.license       = "MIT"
+
+  # Prevent pushing this gem to RubyGems.org. To allow pushes either set the 'allowed_push_host'
+  # to allow pushing to a single host or delete this section to allow pushing to any host.
+  if spec.respond_to?(:metadata)
+    spec.metadata['allowed_push_host'] = "https://rubygems.org"
+  else
+    raise "RubyGems 2.0 or newer is required to protect against public gem pushes."
+  end
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "nokogiri", "~> 1"
+  spec.add_dependency "open_uri_redirections", "~> 0.2"
+  spec.add_dependency "thread", "~> 0.2"
+
+  spec.add_development_dependency "bundler", "~> 1.10"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.0"
+end

metadata

@@ -0,0 +1,145 @@
+--- !ruby/object:Gem::Specification
+name: super_crawler
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Hassen Taidirt
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2016-07-09 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1'
+- !ruby/object:Gem::Dependency
+  name: open_uri_redirections
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+- !ruby/object:Gem::Dependency
+  name: thread
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+description: SuperCrawler allows you to easily crawl full web sites or web pages (extracting
+  internal links and assets) in few seconds.
+email:
+- htaidirt@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- CODE_OF_CONDUCT.md
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lib/super_crawler.rb
+- lib/super_crawler/crawl_page.rb
+- lib/super_crawler/crawl_site.rb
+- lib/super_crawler/version.rb
+- super_crawler.gemspec
+homepage: https://github.com/htaidirt/super_crawler
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: Easy (yet efficient) ruby gem to crawl your favorite website.
+test_files: []