disavow_tool 0.3.2 → 0.3.3s

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 77189b8d9bcfe6f39a02e4f6f9c1029a8e0195db398a77cc3b41710298b29b48
-  data.tar.gz: 7420106da6fe2ff050a538c7d65b76bea2215513a1992d1ec230d8432da52e99
+  metadata.gz: a1c1d4a1ba4b5d7a6898620d390f3c66005a9c3686bef4dcc4fa372eadd2913b
+  data.tar.gz: acc55fe0bb006e74ee177863f6222f7a2d25a9f1f7f7475b50460a85c1c073fa
 SHA512:
-  metadata.gz: bfba74cb97ecbcb18513d916db793aa2504d9c64e3502cfab5719b3452b1ab1d1886532a4370906eaf34b47651169d2ab649a9e958be1af76b62a174e062dee9
-  data.tar.gz: 567a7ea282a3a726eeae9ca838f4223a0d126a3812edb035991ca93d836dfdb8c87375ca056ae31d973be8b2735689dcefdecf33a5bcdc4b5854636aa1a0ef2a
+  metadata.gz: e791192b0ed9d61d1e553acfc3c4e62c56b90acc4f2091381cd6bf464ff12f0db8d46ce1fb7d9f16c6be7951221c53d6e41624595a6942547a647def0ab14f7b
+  data.tar.gz: a2b30d3d21728baa845d843c50d15319203b572610d507cc71ba72500dab23a7419c99336167bb05608f02e7253e7566428f454d824c9b7b191f90cc8b6c86a2

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    disavow_tool (0.3.2)
+    disavow_tool (0.3.3s)
       activesupport
       colorize
       nokogiri

data/README.md

@@ -1,38 +1,141 @@
 # DisavowTool
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/disavow_tool`. To experiment with that code, run `bin/console` for an interactive prompt.
+A simple tool to create a Disavow file to be uploaded to the [Google Disavow Tool](https://www.google.com/webmasters/tools/disavow-links-main). You'll be able to import the current disavow file, a whitelist with all the trusted URLs and domains and analyse them against new imported URLs you feed DisavowTool with. Auditing and cleaning the back links to your site is essential to have a good SEO.
 
-TODO: Delete this and the text above, and describe your gem
+Tips and donations: Bitcoin SV address **1NN3eRawdJ2dPoKQEnHAY7ug3Q6TwbJ4c3**
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Simply install the gem:
 
-```ruby
-gem 'disavow_tool'
-```
+    $ gem install disavow_tool
 
-And then execute:
+You'll have the new command available in your system: `disavow_tool`
 
-    $ bundle
+## Usage
 
-Or install it yourself as:
+From the command's help `disavow_tool --help` option:
 
-    $ gem install disavow_tool
+    Usage: disavow.rb [options] --disavow file_1,file_2,file_3\
+                                --import file_1,file_2,file_3\
+                                [--whitelist file1,file2,file3]
 
-## Usage
+You simply feed DisavowTool with the current disavow file you have and all the new links you haven't analysed yet. You can feed multiple disavow files and import files. Optionally you can feed DisavowTool with a whitelist so that DisavowTool can do the heavy lifting
+
+DisavowTool will try to remove from the imported URLs all known URLS provided in the Disavow files and the Whitelist files as well as removing all URLs with the same domain as found in Disavow or Whitelist.
+
+Once the cleanup of the imported URLs is done you'll be asked URL by URL what DisavowTool should do:
+
+      Links pending to analysed 241
+      **************************************************************
+      *
+      * Analysing url: https://scammysyte.com/wordpress-comments
+      * Website title: Viagra for all
+      * URls with this same domain: 180
+      *
+      ***************************************************************
+      [W] Whitelist as domain [w] Whitelist URL [a] whitelist as url All urls with this domain
+      [D] Disavow as domain [d] Disavow as URL
+      [o] to open the URL [.] to exit
+
+After the analysis is complete, DisavowTool will export a Disavow file ready to me uploaded to the [Google Disavow Tool](https://www.google.com/webmasters/tools/disavow-links-main). DisavowTool will also export a new whitelist, ideally you should feed this whitelist the next time you run your analysis for the same domain.
+
+## Disavow files format
+The format will be the same as [defined by Google](https://support.google.com/webmasters/answer/2648487?hl=en)
+    # Comments
+    # Domains
+    domain:scammysite.com
+    domain:scammysite2.com
+    # URLs
+    https://viagraforall.com/wordpress_comments
+    https://seo_scammers.com/wordpress_comments10
+
+## Whitelist files format
+The format will be the same as [defined by Google](https://support.google.com/webmasters/answer/2648487?hl=en)
+    # Comments
+    # Domains
+    domain:most_popular_site.com
+    domain:wow_this_site_linked_back_to_me.com
+    # URLs
+    https://maybe_a_good_site.com/the_best_product
+    https://should_we_trust_this_domain.com/product_review
+
+## Import files format
+Each line should contain a complete URL.
+
+    https://newsite.com/path
+    https://newsite2.com/path
+    https://newsite3.com/path
+
+
+## FAQ
+
+### Why do I need to keep track of the toxic domains?
+Well... Google lies https://www.youtube.com/watch?v=HWJUU-g5U_I
+
+![alt text](miscellaneous/images/communist.jpg)
+
+The reality is your site will get killed with a negative SEO attack, period.
+
+Just check one of our sites that received an attack:
+
+![alt text](miscellaneous/images/killed2.png)
+
+Google is slow, your site's SEO won't be affected instantly if you are a victim of this type of attack but you have a colony of termites eating you from the inside. One day, out of nowhere your wood will crack and if your revenue is very SEO dependant you may be very close to bankruptcy if not completely broke. You better integrate a backlink clean up in your SEO routie as soon as possible.
+
+In essence you need to do all the work Goole, in our opinion, should be doing.
+
+### Why not just use _SEMrush back link audit_ or _Ahrefs_?
+
+Good question.
+
+Ahrefs doesn't let you import links other than what their software finds so you are out of luck if you have a better source of new backlinks.
+
+SEMrush is better in that regard as they let you import new back links and connect your account to other services such as Google Search Console or services like Majestic. The bad news is SEMrush crawls the web with **curl** identifying themselves against the new imported backlinks and if they get an HTTP error, they discard the site and you aren't able to put that domain or URL into your disavow file from their website application. The result is malicious sites return HTTP 500 or 403 when they detect bots to detect them. Just as example, this clear malicious or infected site http://carreviewauto.com/statti/kak-uberech-avto-ot-ugona-4-zolotyx-pravila.html​ returns 403 for the SEMSush routine. SEMRush's curl command:
+
+      curl -i -sS -L --proto-redir -all,http,https --max-time 5 -A 'Mozilla/5.0 (compatible; SemrushBot-SA/0.97;+http://www.semrush.com/bot.html)' -H 'Accept-Encoding: gzip, deflate' -H'Accept: */*' --compressed http://carreviewauto.com/statti/kak-uberech-avto-ot-ugona-4-zolotyx-pravila.html​
+
+Response headers:
+
+      HTTP/1.1 403 Forbidden
+      Server: nginx
+      Date: Sun, 04 Aug 2019 13:23:04 GMT
+      Content-Type: text/html; charset=iso-8859-1
+      Content-Length: 334
+      Connection: keep-alive
+      Keep-Alive: timeout=60
+
+SEMRush's customer service has declined to change this.
+
+## Our workflow
+1. We keep using SEMRush backlink audit tool as it's a bit more polished. Once a month we run their tool and download the generated disavow file _(disavow_semrush.cvs)_ and whitelist _(whitelist.cvs)_.
+
+2. We also download the current disavow file uploaded to [Google Disavow Tool](https://www.google.com/webmasters/tools/disavow-links-main).
+
+3. We download the most sampled _(GSC_most_sampled.cvs)_ and latest links _(GSC_latest.cvs)_ found by [Google Search Console](https://search.google.com/search-console).
+
+4. We download the new backlinks found by [ahrefs](https://ahrefs.com) _(new_ahrefs.cvs)_
+
+5. We download the new backlinks found by [Majestic](Majestic.com) _(new_majestic.cvs)_
+
+6. We have the whitelist generated by disavow in our last session whitelist_last_disavow_tool_session.csv
+
+7. We run the following command:
+        disavow_tool -V --disavow disavow_semrush.cvs,whitelist.cvs\
+                        --whitelist.cvs,whitelist_last_disavow_tool_session.csv\
+                        --import GSC_most_sampled.cvs,GSC_latest.cvs,new_ahrefs.cvs,new_majestic.cvs
 
-TODO: Write usage instructions here
+8. Upload the generated disavow file to [Google Disavow Tool](https://www.google.com/webmasters/tools/disavow-links-main) and save the generated whitelist file for the next session.
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/disavow_tool.
+Bug reports and pull requests are welcome on GitHub at https://github.com/maesitos/disavow_tool.
 
 ## License
 

data/lib/disavow_tool/command_options.rb

@@ -16,7 +16,7 @@ module DisavowTool
       options.network_requests = true
 
       opt_parser = OptionParser.new do |opts|
-          opts.banner = "Usage: disavow.rb [options] --disavow FILE --import file_1,file_2,file_3 [--whitelist file1,file2,file3]"
+          opts.banner = "Usage: disavow_tool [options] --disavow file_1,file_2,file_3 --import file_1,file_2,file_3 [--whitelist file1,file2,file3]"
           opts.separator ""
           opts.separator "Requited options:"
           opts.on("-d","--disavow file_1,file_2", Array, "Disavow files as exported from Google Search Console") do |file|

data/lib/disavow_tool/list.rb

@@ -9,15 +9,17 @@ module DisavowTool
 
     def initialize(import_files)
       @list = Set.new
+      @list_files_imported = []
       @verbose = OPTIONS.verbose
       @verbose_hard = OPTIONS.hardcore_verbose
-      import import_files if @verbose
+      import import_files
       @original_list = @list.clone
     end
 
     def import(import_files)
       import_files = [import_files] if import_files.class != Array
       import_files.each do |file|
+        @list_files_imported << file
         puts "Importing file: #{file}"
         File.readlines(file).each do |line|
           line.chomp!
@@ -89,6 +91,9 @@ module DisavowTool
     def summary(list=nil, original_list=nil)
       list = list || @list
       original_list = original_list || @original_list
+      puts "Files imported:"
+      puts @list_files_imported
+
       puts "#{message_sumary_imported} #{original_list.count}".blue
       puts "#{mensaje_sumary_before_export} #{list.count}".blue
 

data/lib/disavow_tool/version.rb

@@ -1,3 +1,3 @@
 module DisavowTool
-  VERSION = "0.3.2"
+  VERSION = "0.3.3s"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: disavow_tool
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.3.3s
 platform: ruby
 authors:
 - Maesitos
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2019-08-03 00:00:00.000000000 Z
+date: 2019-08-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -124,6 +124,9 @@ files:
 - lib/disavow_tool/list.rb
 - lib/disavow_tool/version.rb
 - lib/disavow_tool/white_list.rb
+- miscellaneous/images/communist.jpg
+- miscellaneous/images/killed1.png
+- miscellaneous/images/killed2.png
 - samples/disavowed.cvs
 - samples/disavowed2.cvs
 - samples/new_links.csv
@@ -148,9 +151,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubygems_version: 3.0.1
 signing_key: