powerdlz23 1.2.3 → 1.2.4

package/rubyretriever/readme.md

@@ -0,0 +1,166 @@
+RubyRetriever
+==============
+[![Gem Version](https://badge.fury.io/rb/rubyretriever.svg)](http://badge.fury.io/rb/rubyretriever)  [![Build Status](https://travis-ci.org/joenorton/rubyretriever.svg?branch=master)](https://travis-ci.org/joenorton/rubyretriever)  
+
+[RubyRetriever Webpage](https://norton.io/projects/rubyretriever/)
+  
+By Joe Norton  
+
+RubyRetriever is a Web Crawler, Scraper & File Harvester. Available as a command-line executable and as a crawling framework.
+
+RubyRetriever (RR) uses asynchronous HTTP requests via [Eventmachine](https://github.com/eventmachine/eventmachine) & [Synchrony](https://github.com/igrigorik/em-synchrony) to crawl webpages *very quickly*. RR also uses a Ruby implementation of the [bloomfilter](https://github.com/igrigorik/bloomfilter-rb) in order to keep track of pages it has already crawled in a memory efficient manner.  
+
+**v1.4.3 Update (3/24/2016)** - Fixes problem with file downloads that had query strings, the filename was being saved with the querystrings still attached. No more.
+
+**v1.4.2 Update (3/24/2016)** - Fixes problem with named anchors (divs) being counted as links.
+
+**v1.4.1 Update (3/24/2016)** - Update gemfile & external dependency versioning
+
+**v1.4.0 Update (3/24/2016)** - Several bug fixes.
+
+
+Mission  
+-------
+RubyRetriever aims to be the best command-line crawling and scraping package written in Ruby and a replacement for paid software such as Screaming Frog SEO Spider.    
+
+
+Roadmap?  
+Not sure. Feel free to offer your thoughts.  
+
+Some Potential Ideas:  
+* 'freeroam mode' - to go on cruising the net endlessly in fileharvest mode  
+* 'dead-link finder' mode - collects links returning 404, or other error msgs    
+* 'validate robots.txt' mode - outputs the bot-exposed sitemap of your site  
+* more sophisticated SEO analysis? replace screaming frog? this would include checks for canonical URL, maybe some keyword density checks, content length checks, etc.    
+
+Features  
+--------  
+* Asynchronous HTTP Requests thru EM & Synchrony  
+* Bloom filter for tracking visited pages
+* Supports HTTPS  
+* Follows 301 redirects (if to same host)  
+* 3 CLI modes  
+	* Sitemap - Find all links on a website, output a valid XML sitemap, or just a CSV  
+	* File Harvest - find all files linked to on a website, option to autodownload  
+	* SEO  - collect important SEO info from every page, output to a CSV (or STDOUT)  
+* Run a Custom Block on a Per-Page basis (PageIterator)  
+
+Use cases  
+---------
+**As an Executable**  
+With a single command at the terminal, RR can:  
+1. Crawl your website and output a *valid XML sitemap* based on what it found.  
+2. Crawl a target website and *download all files of a given filetype*.  
+3. Crawl a target website, *collect important SEO information* such as page titles, meta descriptions and h1 tags, and write it to CSV.  
+
+**Used in Custom scripts**  
+As of version 1.3.0, with the PageIterator class you can pass a custom block that will get run against each page during a crawl, and collect the results in an array. This means you can define for yourself whatever it is you want to collect from each page during the crawl.  
+
+Help & Forks Welcome!  
+  
+Getting started   
+-----------
+Install the gem
+```sh
+$ gem install rubyretriever
+```  
+  
+
+Using the Executable  
+--------------------
+ **Example: Sitemap mode**  
+```sh
+$ rr --sitemap CSV --progress --limit 10 http://www.cnet.com
+```  
+OR -- SAME COMMAND  
+```sh
+$ rr -s csv -p -l 10 http://www.cnet.com
+```  
+  
+This would map http://www.cnet.com until it crawled a max of 10 pages, then write the results to a CSV named cnet. Optionally, you could also use the format XML and RR would output the same URL list into a valid XML sitemap that could be submitted to Google.  
+  
+ **Example: File Harvesting mode**  
+```sh
+$ rr --files txt --verbose --limit 1 http://textfiles.com/programming/
+```  
+OR -- SAME COMMAND  
+```sh
+$ rr -f txt -v -l 1 http://textfiles.com/programming/
+```  
+  
+This would crawl http://textfiles.com/programming/ looking for txt files for only a single page, then write out a list of filepaths to txt files to the terminal. Optionally, you could have the script autodownload all the files by adding the -a/--auto flag.
+
+**Example: SEO mode**  
+```sh
+$ rr --seo --progress --limit 10 --out cnet-seo http://www.cnet.com
+```  
+OR -- SAME COMMAND  
+```sh
+$ rr -e -p -l 10 -o cnet-seo http://www.cnet.com
+```  
+  
+This would go to http://www.cnet.com and crawl a max of 10 pages, during which it would collect the SEO fields on those pages - this currently means [url, page title, meta description, h1 text, h2 text]. It would then write the fields to a csv named cnet-seo.
+  
+
+command-line arguments
+-----------------------
+Usage: rr [MODE FLAG] [OPTIONS] Target_URL  
+
+Where MODE FLAG is required, and is either:  
+	-s, --sitemap FORMAT  (only accepts CSV or XML atm)  
+	-f, --files FILETYPE  
+	-e, --seo  
+  
+and OPTIONS is the applicable:  
+    -o, --out FILENAME                  *Dump fetch data as CSV*  
+    -p, --progress						*Outputs a progressbar*  
+    -v, --verbose                       *Output more information*  
+    -l, --limit PAGE_LIMIT_#            *set a max on the total number of crawled pages*  
+    -h, --help                          *Display this screen*  
+  
+
+Using as a Library (starting as of version 1.3.0)  
+------------------
+
+If you want to collect something, other than that which the executable allows, on a 'per page' basis then you want to use the PageIterator class. Then you can run whatever block you want against each individual page's source code located during the crawl.   
+
+Sample Script using **PageIterator**  
+```ruby
+require 'retriever'
+opts = {
+  'maxpages' => 1
+}
+t = Retriever::PageIterator.new('http://www.basecamp.com', opts) do |page|
+  [page.url, page.title]
+end
+puts t.result.to_s
+```
+
+```sh
+>> [["http://www.basecamp.com", "Basecamp is everyone’s favorite project management app."]]  
+```  
+Available methods on the page iterator:  
+* **#url** - returns full URL of current page  
+* **#source** - returns raw page source code  
+* **#title** - returns html decoded verson of curent page title  
+* **#desc** - returns html decoded verson of curent page meta description  
+* **#h1**  - returns html decoded verson of current page's h1 tag  
+* **#h2**  - returns html decoded verson of current page's h2 tag
+* **#links** - returns array of all links on the page  
+* **#parse_internal** - returns array of current page's internal (same host) links  
+* **#parse_internal_visitable** - returns #parse_internal plus added filtering of only links that are visitable  
+* **#parse_seo** - returns array of current page's html decoded title, desc, h1 and h2  
+* **#parse_files** - returns array of downloaded files of type supplied as RR options (fileharvest options)  
+
+
+Current Requirements
+------------ 
+em-synchrony  
+ruby-progressbar  
+bloomfilter-rb  
+addressable  
+htmlentities  
+
+License
+-------  
+See included 'LICENSE' file. It's the MIT license.

package/rubyretriever/rubyretriever.gemspec

@@ -0,0 +1,41 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'retriever/version'
+
+Gem::Specification.new do |s|
+  s.required_ruby_version = ['>= 2.0', '< 2.3']
+  s.platform    = Gem::Platform::RUBY
+  s.version     = Retriever::VERSION
+  s.name        = 'rubyretriever'
+  s.date        = '2016-04-11'
+  s.summary     = 'Ruby Web Crawler & File Harvester'
+  s.description = 'Asynchronous web crawler, scraper and file harvester'
+  s.authors     = ['Joe Norton']
+  s.email       = ['joe@norton.io']
+  s.homepage    = 'http://norton.io/rubyretriever/'
+  s.license     = 'MIT'
+  # If you need to check in files that aren't .rb files, add them here
+  s.files        = Dir['{lib}/**/*.rb', 'bin/*', 'LICENSE', '*.md',
+                       '{spec}/*.rb']
+  s.require_path = 'lib'
+  s.rubyforge_project         = 'rubyretriever'
+
+  # If you need an executable, add it here
+  s.executables = ['rr']
+  s.required_rubygems_version = '>= 1.3.6'
+
+  # If you have other dependencies, add them here
+  s.add_runtime_dependency 'em-synchrony'
+  s.add_runtime_dependency 'em-http-request'
+  s.add_runtime_dependency 'ruby-progressbar'
+  s.add_runtime_dependency 'bloomfilter-rb'
+  s.add_runtime_dependency 'addressable'
+  s.add_runtime_dependency 'htmlentities'
+  s.add_runtime_dependency 'nokogiri'
+
+  s.add_development_dependency 'bundler', '~> 1.6'
+  s.add_development_dependency 'rake', '~> 10.3'
+  s.add_development_dependency 'rspec', '~> 2.14'
+  s.add_development_dependency 'pry'
+end

package/rubyretriever/spec/link_spec.rb

@@ -0,0 +1,77 @@
+require 'retriever'
+
+describe 'Link' do
+
+  t = Retriever::Target.new('http://www.cnet.com/reviews/')
+  let(:links) do
+    Retriever::Page.new('http://www.cnet.com/reviews/', @source, t).links
+  end
+
+  it 'collects links in anchor tags' do
+    @source = (<<SOURCE).strip
+    <a href='http://www.cnet.com/download.exe'>download</a>
+SOURCE
+
+    expect(links).to include('http://www.cnet.com/download.exe')
+  end
+
+  it 'collects links in link tags' do
+    @source = (<<SOURCE).strip
+<link rel='stylesheet' id='gforms_reset_css-css'  href='http://www.cnet.com/wp-content/plugins/gravityforms/css/formreset.css?ver=1.7.12' type='text/css' media='all' />
+SOURCE
+
+    expect(links[0]).to include('formreset.css?ver=1.7.12')
+  end
+
+  it 'does not collect bare links (ones not in an href)' do
+    @source = (<<SOURCE).strip
+http://www.google.com
+SOURCE
+
+    expect(links).to_not include('http://www.google.com')
+  end
+
+  it 'collects only unique href links on the page' do
+    @source = (<<SOURCE).strip
+<a href='http://www.cnet.com/products/gadgets'>gadgets</a>
+<a href='http://www.cnet.com/products/gadgets'>gadgets2</a>
+SOURCE
+
+    expect(links.size).to eq(1)
+  end
+
+  it 'adds a protocol to urls missing them (www.)' do
+    @source = (<<SOURCE).strip
+<a href='www.cnet.com/download.exe'>download</a>
+SOURCE
+
+    expect(links).to include('http://www.cnet.com/download.exe')
+  end
+
+  it "doesn\'t care about any extra attributes on the anchor tag" do
+    @source = (<<SOURCE).strip
+<a href='http://www.cnet.com/products/gadgets/'>gadgets </a>
+<a href='http://www.cnet.com/products/gadgets/' data-vanity-rewritten='true'>
+</a>
+SOURCE
+
+    expect(links.size).to eq(1)
+  end
+
+  it 'returns relative urls with full path based on hostname' do
+    @source = (<<SOURCE).strip
+<a href='/test.html'>test</a>
+<a href='cpage_18'>about</a>
+SOURCE
+
+    expect(links).to include('http://www.cnet.com/test.html',
+                             'http://www.cnet.com/reviews/cpage_18')
+  end
+  it 'collects files even when query strings exist' do
+    @source = (<<SOURCE).strip
+    <a href='http://mises.org/system/tdf/Robert%20Nozick%20and%20Murray%20Rothbard%20David%20Gordon.mp3?file=1&amp;type=audio' type='audio/mpeg; length=22217599' title='Robert Nozick and Murray Rothbard David Gordon.mp3'>Download audio file</a></span></div>
+SOURCE
+
+    expect(links).to include('http://mises.org/system/tdf/Robert%20Nozick%20and%20Murray%20Rothbard%20David%20Gordon.mp3?file=1&amp;type=audio')
+  end
+end

package/rubyretriever/spec/page_spec.rb

@@ -0,0 +1,94 @@
+require 'retriever/page'
+require 'retriever/fetch'
+
+t = Retriever::Target.new('http://www.cnet.com/reviews/', /\.exe\z/)
+
+describe 'Page' do
+  let(:common_source) do
+    <<-SOURCE
+    <title>test</title>
+    <a href='www.cnet.com/download.exe'>download</a>
+    <a href='/test.html'>test</a>
+    <a href='http://www.cnet.com/products/gadgets/' data-vanity-rewritten='true'>
+    </a>
+    <a href='http://www.cnet.com/products/gadgets/' id='gadgets-link'>gadgets </a>
+    <a href='http://www.yahoo.com/test/'>yahoo</a>"
+    <meta name='description' content="test2 ">
+    <h1>test 3</h1>
+    <h2> test 4 </h2>
+    SOURCE
+  end
+
+  describe '#url' do
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', common_source, t) }
+    it 'returns current page URL' do
+      expect(page.url).to eq('http://www.cnet.com/')
+    end
+  end
+
+  describe '#links' do
+    let(:source) { "<a href='/profile/'>profile</a><a href='#top'>top</a> <link rel='stylesheet' id='gforms_reset_css-css'  href='http://www.cnet.com/wp-content/plugins/gravityforms/css/formreset.css?ver=1.7.12' type='text/css' media='all' />" }
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', source, t) }
+    it 'collects all unique href links on the page, skips div anchors' do
+      expect(page.links.size).to eq(2)
+    end
+  end
+
+  describe '#parse_internal' do
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', common_source, t) }
+    let(:links) { page.parse_internal }
+    it 'filters links by host' do
+      expect(links.size).to eq(3)
+    end
+  end
+
+  describe '#parse_internal_visitable' do
+    let(:source) { "<a href='/profile/'>profile</a> <link rel='stylesheet' id='gforms_reset_css-css'  href='http://www.cnet.com/wp-content/plugins/gravityforms/css/formreset.css?ver=1.7.12' type='text/css' media='all' />" }
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', source, t) }
+    let(:links) { page.parse_internal_visitable }
+    it "filters out 'unvisitable' URLS like JS, Stylesheets, Images" do
+      expect(links.size).to eq(1)
+    end
+  end
+
+  describe '#parse_files' do
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', common_source, t) }
+    let(:files) { page.parse_files(page.parse_internal) }
+    it 'filters links by filetype' do
+      expect(files.size).to eq(1)
+    end
+  end
+
+  describe '#parse_by_css' do
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', common_source, t) }
+
+    it 'returns the text from the received css selector' do
+      expect(page.parse_by_css('#gadgets-link')).to eq('gadgets ')
+    end
+  end
+
+  describe '#title' do
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', common_source, t) }
+    it 'returns page title' do
+      expect(page.title).to eq('test')
+    end
+  end
+  describe '#desc' do
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', common_source, t) }
+    it 'returns meta description' do
+      expect(page.desc).to eq('test2 ')
+    end
+  end
+  describe '#h1' do
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', common_source, t) }
+    it 'returns h1 text' do
+      expect(page.h1).to eq('test 3')
+    end
+  end
+  describe '#h2' do
+    let(:page) { Retriever::Page.new('http://www.cnet.com/', common_source, t) }
+    it 'returns h2 text' do
+      expect(page.h2).to eq(' test 4 ')
+    end
+  end
+end

package/rubyretriever/spec/retriever_spec.rb

@@ -0,0 +1,84 @@
+require 'retriever'
+require 'retriever/fetchfiles'
+
+describe 'Fetch' do
+  let(:r) do
+    Retriever::Fetch.new('http://www.yahoo.com', {})
+  end
+  describe '#good_response?' do
+
+    let(:resp) do
+      {}
+    end
+
+    let(:nil_response) do
+      r.good_response?(nil, 'http://www.yahoo.com')
+    end
+
+    let(:unsuccessful_resp) do
+      resp.stub(:response_header).and_return(resp)
+      resp.stub(:redirection?).and_return(false)
+      resp.stub(:successful?).and_return(false)
+      resp.stub(:server_error?).and_return(false)
+      resp.stub(:client_error?).and_return(false)
+      r.good_response?(resp, 'http://www.yahoo.com')
+    end
+
+    let(:redir_resp) do
+      resp.stub(:response_header).and_return(resp)
+      resp.stub(:redirection?).and_return(true)
+      resp.stub(:location).and_return('http://www.google.com')
+      r.good_response?(resp, 'http://www.yahoo.com')
+    end
+
+    let(:bad_content_type_resp) do
+      resp.stub(:response_header).and_return(resp)
+      resp.stub(:redirection?).and_return(false)
+      resp.stub(:successful?).and_return(true)
+      resp['CONTENT_TYPE'] = 'image/jpeg'
+      r.good_response?(resp, 'http://www.yahoo.com')
+    end
+
+    let(:success_resp) do
+      resp.stub(:response_header).and_return(resp)
+      resp.stub(:redirection?).and_return(false)
+      resp.stub(:successful?).and_return(true)
+      resp['CONTENT_TYPE'] = 'text/html'
+      r.good_response?(resp, 'http://www.yahoo.com')
+    end
+
+    it 'returns false if the response is empty' do
+      expect(nil_response).to eq(false)
+    end
+
+    it 'returns false on unsuccessful connection' do
+      expect(unsuccessful_resp).to eq(false)
+    end
+
+    it 'returns false on redirecting host' do
+      expect(redir_resp).to eq(false)
+    end
+
+    it 'returns false on non-visitable content type' do
+      expect(bad_content_type_resp).to eq(false)
+    end
+
+    it 'returns true otherwise' do
+      expect(success_resp).to eq(true)
+    end
+  end
+  describe '#filter_out_querystrings' do
+    let(:normal_url) do
+      r.filter_out_querystrings('http://mises.org/test.mp3')
+    end
+    let(:query_string_url) do
+      r.filter_out_querystrings('http://mises.org/system/tdf/Robert%20Nozick%20and%20Murray%20Rothbard%20David%20Gordon.mp3?file=1&type=audio')
+    end
+    it 'accepts standard urls' do
+      expect(normal_url).to eq('http://mises.org/test.mp3')
+    end
+    it 'strips query params' do
+      expect(query_string_url).to eq('http://mises.org/system/tdf/Robert%20Nozick%20and%20Murray%20Rothbard%20David%20Gordon.mp3')
+    end
+  end
+end

package/rubyretriever/spec/spec_helper.rb

@@ -0,0 +1,17 @@
+# This file was generated by the `rspec --init` command. Conventionally, all
+# specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`.
+# Require this file using `require "spec_helper"` to ensure that it is only
+# loaded once.
+#
+# See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration
+RSpec.configure do |config|
+  config.treat_symbols_as_metadata_keys_with_true_values = true
+  config.run_all_when_everything_filtered = true
+  config.filter_run :focus
+
+  # Run specs in random order to surface order dependencies. If you find an
+  # order dependency and want to debug it, you can fix the order by providing
+  # the seed, which is printed after each run.
+  #     --seed 1234
+  config.order = 'random'
+end

package/rubyretriever/spec/target_spec.rb

@@ -0,0 +1,55 @@
+require 'retriever'
+require 'open-uri'
+
+describe 'Target' do
+  let(:t) do
+    Retriever::Target.new('http://www.cnet.com/reviews/', /\.exe\z/)
+  end
+
+  it 'creates target var' do
+    expect(t.target).to eq('http://www.cnet.com/reviews/')
+  end
+
+  it 'creates host var' do
+    expect(t.host).to eq('www.cnet.com')
+  end
+
+  it 'creates host_re var' do
+    expect(t.host_re).to eq(/cnet.com/)
+  end
+
+  it 'creates port var (no port specified)' do
+    expect(t.port).to be_nil
+  end
+
+  it 'creates port var (with port specified)' do
+    expect(Retriever::Target.new('http://www.cnet.com:3000/reviews/', /\.exe\z/).port).to be(3000)
+  end
+
+  it 'creates file_re var (when provided)' do
+    expect(t.file_re).to eq(/\.exe\z/)
+  end
+
+  it 'adds protocol to Target URL if none given' do
+    expect(Retriever::Target.new('cnet.com').target).to eq('http://cnet.com')
+  end
+
+  it 'fails if given URL has no dot in it' do
+    expect { Retriever::Target.new('cnetcom') }.to raise_error
+  end
+
+  describe '#source' do
+    let(:redirecting_url) do
+      Retriever::Target.new('http://software-by-joe.appspot.com').source
+    end
+
+    it 'opens URL and returns source as String' do
+      expect(Retriever::Target.new('http://techcrunch.com/').source.class)
+      .to eq(String)
+    end
+
+    it 'fails if target redirects to new host' do
+      expect { redirecting_url }.to raise_error
+    end
+  end
+end