crm_formatter 1.0.6.pre.rc.1 → 1.0.7.pre.rc.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '031873afa102ba537d240f6c86c828bd29c75d85bf5063af6f6f5b715398203f'
-  data.tar.gz: a4c08972f40519a94ac636b5c5f287c68bb485bb6ae3c1126ffcc80a61dd3b2a
+  metadata.gz: c1a9e605ef7ec90b0c88e80a32cd22fa604a9cc810fe559eb19c389c72c0e494
+  data.tar.gz: 07b4436cf1d31125e44a0bd71424b7c7c21a50954faa8e14b2792f08fd7391d9
 SHA512:
-  metadata.gz: 6a37c47288003b2593422fc199d6875dfb47a0104d48d53b1bb2366a59c7038229082cb17d5b1adc14b14f4227d704c07cfd6672adb5678c7b7744a0c6fb8169
-  data.tar.gz: 4ca91da1f810d5d93451fd6d36d82878185fc33df5f32e7318537a4fa5367eb9cf91ac22c038261bbb38340cf2e207424ae9f68175ba7c80174a34e3f0662cbd
+  metadata.gz: 26357b48e1933ed4a421c9fc8ce1bd01a097d553e7d1810910bb7b891496f341ce456a45cff6313189adf1321e7fe95aad25bf9d4bf6471301dfa61bf4c637c3
+  data.tar.gz: bb86e8cdcc5ea5526e16d7963687e4dd058e3097bdc2b88f6b65284f5cd282ab7f6890304abf90477e1f7503c0a0023b89f48b41c994467bc0f52feffc8509b5

data/.gitignore

@@ -11,3 +11,5 @@ crm_formatter-*.gem
 .DS_Store
 .idea/
 .txt
+.csv
+!extensions.csv

data/README.md

@@ -1,14 +1,21 @@
-# CRMFormatter
 
-Reformat and Normalize CRM Contact Data, Addresses, Phones, Emails and URLs.  
-Please note, that this gem is a rapid work in process.  It is from a collection of modules currently being used on a production app, but decided to open them up.  The tests have not yet been written for the gem and there will still be changes in the near future.  Documentation is limited, but is coming.  Here are some basic points below to help you get started.
+# **CRM Formatter**
+
+#### Reformat and Normalize CRM Contact Data, such as Addresses, Phones and URLs.
+
+**CRM Formatter** was originally designed to curate high-volume enterprise-scale asynchronous web scraping via Nokogiri, Mechanize, and Delayed_job.  Web Scraping *aka Web Harvesting / Data Mining* is notoriously unreliable *sticky* work with endless edge-cases to overcome.  Accurately, yet efficiently curating such data is a constant and evolving task, and will continue to be the core functionality of **CRM Formatter**.
+However, it also plays an integral role in routine functions of apps, like formatting, normalizing, and even scrubbing existing databases, and submitted form data before saving to the database; via model callbacks, such as `before_validation` or `before_save`.
+
+###### The **CRM Formatter** Gem is currently in `--pre` versioning, aka **beta mode** with frequent updates. Formal tests in the gem environment are still on the way.
+However, **CRM Formatter** has been developed continuously for several years and is a reliable and integral part of a production CRM data verification app.  The process of isolating the various modules into a consolidated open source gem has just recently begun, so documentation is still limited, but is frequently being added and refined.
 
 ## Getting Started
+**CRM Formatter** is compatible with Rails 4.2 and 5.0, 5.1 and 5.2 on Ruby 2.2 and later.
 
 In your Gemfile add:
 
 ```
-gem 'crm_formatter', '~> 1.0.5.pre.rc.1'
+gem 'crm_formatter', '~> 1.0.6.pre.rc.1'
 ```
 
 Or to install locally:
@@ -17,42 +24,99 @@ Or to install locally:
 gem install crm_formatter --pre
 ```
 
-### Prerequisites
-
-CRMFormatter is optimized for Rails 5.2 and Ruby 2.5, but has worked well in older versions too.
-
-### Architecture
-
-CRMFormatter is the top level Module wrapper comprising of 3 Classes, Address, Phone and Web.  There is also a Helpers Module for shared tasks.
-
-
-### Usage
-
-CRMFormatter could be used anywhere in an app for various reasons.  Most commonly from an app's helper methods, controllers or model.  If you wanted to ensure all new data is properly formatted before saving, methods could be accessed from the model via before_save, callbacks, or validations.  In addition to CRMFormatter formatting CRM data, it also provides detailed reports on what if anything has changed and it always preserves the original data submitted.  Here are a few examples below of how it could be used.
-
-The gem can be used both en mass to clean up an entire database, or fully-integrated into an app's regular environment.
-They were however, designed for an app that Harvests business data for sales and marketing teams, so they work perfectly with NokoGiri and Mechanize!
-
-** These are just examples below, not strict usage guides ...
+## Usage
+
+##### Usage is organized into three sections, Overview, Methods and Examples.
+
+### I. Overview
+
+#### 1. Access and Integration
+##### Using **CRM Formatter** in your app is very simple, and could be accessed from your app's concerns, controllers, helpers, lib, models, or services, but depends on the scope, location, and size of your application and server.    
+  * Simple form submission validations: model callback typically ideal.
+  * Database normalizing tasks: wrapper method in concerns, helpers, or lib typically ideal.
+  * Long running processes like web scraping or high volume APIs calls, like Google Linkedin, or Twitter: the lib or services might be ideal (multithreaded asynchronously even better)
+
+#### 2. Hash Response
+##### Formatted data will always be returned as a hash datatype the following key-value pairs:
+  * The originally submitted data as the first pair.
+  * Formatted data in the remaining pairs.
+  * A T/F boolean indicator pair regarding if the original and formatted data are different.
+
+#### 3. Optional Arguments *OA*
+##### A class can be instantiated with optional arguments *OA*.
+  * OA house the criteria by which you'd like to scrub your data.  
+  * Each is either 'Pos' or 'Neg', for more accurate reporting of your scrubbing results.
+  * List of available Web OA is below, and each accepts data in the hash datatype, aka 'keyword-args'.
+  * For example, you might want to know which URLs contain 'twitter', 'facebook', or 'linkedin' either to focus on developing a list of business social media links, or perhaps you want to use such a list to better avoid such links.
+  * *OA is currently only available for the Web class.*
+  * *Address OA & Phone OA will be available in a future release.*
+
+### II. Methods
+##### CRM Formatter**'s top level module is `CRMFormatter` and contains the following three classes:
+1. Address:  `CRMFormatter::Address.new`
+2. Phone:  `CRMFormatter::Address.new`
+3. Web:  `CRMFormatter::Address.new`
+
+###### Then assign the above to a variable name of your choosing.
+`addr_formatter = CRMFormatter::Address.new`
+`@addr_formatter = CRMFormatter::Address.new`
+
+###### Web accepts optional arguments *OA* as a Hash (with Key-Value pairs)
+Without OA: Instantiate normally if not using OA.
+`web_formatter = CRMFormatter::Web.new`
+
+With OA: Follow the steps to use Web OA:
+1. Available Web OA and the required Key-Value naming and datatypes.  
+* Only list the OA K-V Pairs you're using.  No need to list empty values.  It's not all or nothing. These are empty to illustrate the expected datatypes.
+
+Below is how the OA are received in the Web class at initialization.
+**3. Web Examples at the very bottom has a very detailed example including how OA can be used.**
+```
+def initialize(args={})
+  @empty_oa = args.empty?
+  @pos_urls = args.fetch(:pos_urls, [])
+  @neg_urls = args.fetch(:neg_urls, [])
+  @pos_links = args.fetch(:pos_links, [])
+  @neg_links = args.fetch(:neg_links, [])
+  @pos_hrefs = args.fetch(:pos_hrefs, [])
+  @neg_hrefs = args.fetch(:neg_hrefs, [])
+  @pos_exts = args.fetch(:pos_exts, [])
+  @neg_exts = args.fetch(:neg_exts, [])
+  @min_length = args.fetch(:min_length, 2)
+  @max_length = args.fetch(:max_length, 100)
+end
+```
 
-### Usage by Class & Methods
-* The examples at the bottom of the page are rather verbose, so first, here is a list of methods available to you.
+Example: Below is the syntax for how to use OA.
+There are both Positive and Negative.  They work the same and could just be included in the same array if you prefer.  But they are intended to help you scrub data against negative criteria and for positive criteria.
+```
+oa_args = { neg_urls: %w(approv insur invest loan quick rent repair),
+            neg_links: %w(buy call cash cheap click gas insta),
+            neg_hrefs: %w(after anounc apply approved blog buy call click),
+            neg_exts: %w(au ca edu es gov in ru uk us),
+            min_length: 0,
+            max_length: 30
+          }
+
+@web_formatter = CRMFormatter::Web.new(oa_args)
+```
 
-# Address
+#### 1. Address Methods
 
-* 'get_full_address' takes a hash of address parts then runs each through their respective formatters, then also adds an additional feature of combining them into a long full address string, and indicates if there were any changes from the original version and newly formatted.
+`get_full_address()` takes a hash of address parts then runs each through their respective formatters, then also adds an additional feature of combining them into a long full address string, and indicates if there were any changes from the original version and newly formatted.
 
 ```
   addr_formatter = CRMFormatter::Address.new
 
-  addr_formatter.get_full_address(full_address_hsh)
-  full_address_hsh = {street: street, city: city, state: state, zip: zip}
+  full_address_hash = {street: street, city: city, state: state, zip: zip}
 
-  addr_formatter.format_street(street)
+  addr_formatter.get_full_address(full_address_hash)
 
-  addr_formatter.format_city(city)
+  addr_formatter.format_street(street_string)
 
-  addr_formatter.format_state(state)
+  addr_formatter.format_city(city_string)
+
+  addr_formatter.format_state(state_string)
 
   addr_formatter.format_zip(zip)
 
@@ -60,12 +124,11 @@ They were however, designed for an app that Harvests business data for sales and
 
   addr_formatter.compare_versions(original, formatted)
 
-
 ```
 
-# Phone
+#### Phone Methods
 
-* Subtle but important distinction between 'format_phone' which simply puts a phone in any format, like 555-123-4567 into normalized (555) 123-4567, and 'validate_phone' which also uses 'format_phone' to normalize its output, but is mainly tasked with determining if the phone number seem legitimate. If you know for sure that it is a phone number, but just want to normalize then first try format_phone.  If you are doing web scraping or throwing in strings of text mixed with phones, then validate_phone might work better.
+Subtle but important distinction between 'format_phone' which simply puts a phone in any format, like 555-123-4567 into normalized (555) 123-4567, and 'validate_phone' which also uses 'format_phone' to normalize its output, but is mainly tasked with determining if the phone number seem legitimate. If you know for sure that it is a phone number, but just want to normalize then first try format_phone.  If you are doing web scraping or throwing in strings of text mixed with phones, then validate_phone might work better.
 
 ```
   ph_formatter = CRMFormatter::Phone.new
@@ -76,9 +139,7 @@ They were however, designed for an app that Harvests business data for sales and
 
 ```
 
-# Web
-
-
+#### Web Methods
 
 ```
   web_formatter = CRMFormatter::Web.new
@@ -95,22 +156,12 @@ They were however, designed for an app that Harvests business data for sales and
 
 ```
 
+### III. Examples
+Some of the examples are excessively verbose to help illustrate the datatypes and processes.  Here are a few guidelines and tips:
+**3. Web Examples at the very bottom is the most detailed and recent.  It might be a good place to start.**
+*These are just examples below, not strict usage guides ...*
 
-
-
-
-
-
-
-
-
-* Data will always be returned as hashes, with your original, modified, and details about what has changed.
-* You may pass optional arguments at initialization to provide lists of data to match against, for example a list of words that if in a URL, it would automatically report as junk (but still keeping your original in tact.)
-
-Typically, you will want to create your own method as a wrapper for the Gem methods, like below...
-
-## Address
-** The examples below are rather verbose, so you can make them much more compact of course.
+#### 1. Address Examples
 
 ```
 def self.run_adrs
@@ -132,9 +183,7 @@ end
 
 ```
 
-
-
-## Phone
+#### 2. Phone Examples
 
 In the phone example, format_all_phone_in_my_db could be a custom wrapper method, which when called by Rails C or from a front end GUI process, could grab all phones in db meeting certain criteria to be scrubbed. The results will always be in hash format, such as below.... phone_hash  
 
@@ -153,140 +202,99 @@ end
 phone_hash = { phone: 555-123-4567, valid_phone: (555) 123-4567, phone_edit: true }
 ```
 
+#### 3. Web Examples
 
-## Web
-
-In the example below, you might write a wrapper method named anything you like, such as 'format_a_url' and 'clean_many_websites' to pass in urls to be formatted.
+The steps below will show you an option for how you could integrate larger processes in your app.  
+1. Create a wrapper method you can call from an action or Rails C.  In this example, a new class was also created in Lib for that purpose, as there could be related methods to create.
+* These examples only include `CRMFormatter::Web.new.format_url(url)` method.  There are several additional methods available to you.  Documentation is on the way, but in the mean time, try out the below example, then play around with the others too.   
 
 ```
-  @web = CRMFormatter::Web.new
+# /app/lib/start_crm.rb
 
-  def format_a_url(url)
-    hsh = @web.format_url(url)
-  end
+class StartCrm
 
-  def clean_many_websites(array_of_urls)
+  ##Rails C: StartCrm.run_webs
+  def self.run_webs
+    oa_args = get_args
+    web = CRMFormatter::Web.new(oa_args)
 
-    hashes = array_of_urls.map do |url|
-                @web.format_url(url)
-              end
+    formatted_url_hashes = get_urls.map do |url|
+      url_hash = web.format_url(url)
+    end
 
+    formatted_url_hashes
   end
-```
-
-
-### Additional Usage Examples
-
-### Webs
-* Another example below.
 
+end
 ```
-def self.run_webs
-
-  url_flags = %w(approv avis budget business collis eat enterprise facebook financ food google gourmet hertz hotel hyatt insur invest loan lube mobility motel motorola parts quick rent repair restaur rv ryder service softwar travel twitter webhost yellowpages yelp youtube)
-
-  link_flags = %w(: .biz .co .edu .gov .jpg .net // anounc book business buy bye call cash cheap click collis cont distrib download drop event face feature feed financ find fleet form gas generat graphic hello home hospi hour hours http info insta)
-
-  href_flags = %w(? .com .jpg @ * after anounc apply approved blog book business buy call care career cash charit cheap check click)
+2. Make sure to modify your application config file to recognize your new class.
 
-  extension_flags = %w(au ca edu es gov in ru uk us)
-
-  args = { url_flags: url_flags, link_flags: link_flags, href_flags: href_flags, extension_flags: extension_flags }
-  web = CRMFormatter::Web.new(args)
+```
+#/app/config/application.rb
 
-  urls = Accounts.where.not(url: nil).pluck(:url)
+config.eager_load_paths << Rails.root.join('lib/**')
+config.eager_load_paths += Dir["#{config.root}/lib/**/"]
+```
+3. Create your db query or put together a list of URLs to process, along with any OA to include.  The below example is very verbose, but designed to be helpful. In reality, you might have various criteria saved in the db rather than writing it out.
+In this example, we have auto dealer URLs.  In this process, we're focusing on franchise dealers.
+```
+def self.get_args
+  neg_urls = %w(approv avis budget collis eat enterprise facebook financ food google gourmet hertz hotel hyatt insur invest loan lube mobility motel motorola parts quick rent repair restaur rv ryder service softwar travel twitter webhost yellowpages yelp youtube)
 
-  validated_url_hashes = urls.map { |url| web.format_url(url) }
-  valid_urls = validated_url_hashes.map { |hsh| hsh[:valid_url] }.compact
-  extracted_link_hashes = urls.map { |url| web.extract_link(url) }
-  links = extracted_link_hashes.map { |hsh| hsh[:link] }.compact
-  validated_link_hashes = links.map { |link| web.remove_invalid_links(link) }
-  hrefs = ["Hot Inventory", "Join Our Sale!", "Don't Wait till Later", "Apply Today!", "No Cash Down!"]
-  validated_href_hashes = hrefs.map { |href| web.remove_invalid_hrefs(href) }
+  pos_urls = ["acura", "alfa romeo", "aston martin", "audi", "bmw", "bentley", "bugatti", "buick", "cdjr", "cadillac", "chevrolet", "chrysler", "dodge", "ferrari", "fiat", "ford", "gmc", "group", "group", "honda", "hummer", "hyundai", "infiniti", "isuzu", "jaguar", "jeep", "kia", "lamborghini", "lexus", "lincoln", "lotus", "mini", "maserati", "mazda", "mclaren", "mercedes-benz", "mitsubishi", "nissan", "porsche", "ram", "rolls-royce", "saab", "scion", "smart", "subaru", "suzuki", "toyota", "volkswagen", "volvo"]
 
+  neg_exts = %w(au ca edu es gov in ru uk us)
+  oa_args = {neg_urls: neg_urls, pos_urls: pos_urls, neg_exts: neg_exts}
 end
 
+def self.get_urls
+  urls = ["https://www.stevXXXXXXmitsubishiserviceandpartscenter.com", "https://www.perXXXXXXchryslerjeepcenterville.com", "http://www.peXXXXXXchryslerjeepcenterville.com", "http://www.colXXXXXXchryslerdodgejeepram.com"]
+end
 ```
+4. Run your class and wrapper method in Rails C.  By creating the wrapper method, you have set up the entire process to run like a runner.  In reality, you might have several different criteria accessible from a GUI or even running in Cron Jobs.
 
-### Fully Integrated into an App Example
-** The gem is currently being used within another app in the following way...
+`2.5.1 :001 > StartCrm.run_webs`
 
-```
-  @web_formatter.convert_to_scheme_host(url)
-  @web_formatter.format_url(url)
-```
- The two methods above, which are many available to you in the gem are being used below.
-```
-  curl_result[:response_code] = result&.response_code.to_s
-  web_hsh = @web_formatter.format_url(result&.last_effective_url)
+5. Results are always in a Hash, like below.  The URLs are slightly obfuscated out of respect (it's not a bug).  These are examples from a large DB that runs on a loop 24/7 and gets to each organization about once a week, so it's already pretty well up to date, so there aren't any big changes below, but there are still a few things to point out.
 
-  if web_hsh[:formatted_url].present?
-    curl_result[:verified_url] = @web_formatter.convert_to_scheme_host(web_hsh[:formatted_url])
-  end
+* `:is_reformatted` indicates T/F if url_path and `:formatted_url` differ.  If False, then it means they are the same, or the `:url_path` had significant errors which prevented it from being formatted, thus `:formatted_url` would be nil in such a case.  The reality is that you might have some URLs that are so far off that, that they can't be reliably reformatted, so better to only let them pass if we are confident that they are reliable.
 
-```
+* `:url_path` is the url originally submitted by the client.  It can include directory links on the end too, '/careers/, '/about-us/', etc.
 
-# Above is an isolated sliver of the larger environment shown below...
+* `:formatted_url` is the formatted version of `:url_path`.  It will be stripped of additional paths, '/deals/', '/staff/', etc.  Also, often times people ommit 'http://:' and 'www' in CRMs.  This can sometimes cause errors for users or Mechanized Web Scrapers.  So, those will always be included to ensure consistency.  In our production app we follow up the formatting with url redirect following, which our configurations require the entire path, so it will always be included.  The redirect following gem is already being worked on and will be released as an additional gem shortly.
 
-```
-  def start_curl(url, timeout)
-
-    curl_result = { verified_url: nil, response_code: nil, curl_err: nil }
-    if url.present?
-      result = nil
-
-      begin # Curl Exception Handling
-        begin # Timeout Exception Handling
-          Timeout.timeout(timeout) do
-            puts "\n\n=== WAITING FOR CURL RESPONSE ==="
-            result = Curl::Easy.perform(url) do |curl|
-              curl.follow_location = true
-              curl.useragent = "curb"
-              curl.connect_timeout = timeout
-              curl.enable_cookies = true
-              curl.head = true #testing - new
-            end # result
-
-            curl_result[:response_code] = result&.response_code.to_s
-            web_hsh = @web_formatter.format_url(result&.last_effective_url)
-
-            if web_hsh[:formatted_url].present?
-              curl_result[:verified_url] = @web_formatter.convert_to_scheme_host(web_hsh[:formatted_url])
-            end
-          end
-
-        rescue Timeout::Error # Timeout Exception Handling
-          curl_result[:curl_err] = "Error: Timeout"
-        end
-
-      rescue LoadError => e  # Curl Exception Handling
-        curl_err = error_parser("Error: #{$!.message}")
-        # CheckInt.new.check_int if curl_err.include?('TCP')
-        curl_result[:curl_err] = curl_err
-      end
-    else ## If no url present?
-      curl_result[:curl_err] = 'URL Nil'
-    end
+* `:neg` is an array of all the errors and negative, undesirable criteria to scrub against.  If you include the criteria in OA `neg_urls:`, like above, it will automatically scrub and report.  Regardless, any errors will also be included in there.  So, if the url was not ultimately formatted, there will be details regarding why in `:neg`.
 
-    print_result(curl_result)
-    curl_result
-  end
-```
-
-
-### Another Example from a Production Environment...
+* `:pos` is the opposite, which highlights positive criteria you might be looking for.  It too is available in OA via `pos_urls:`, like above.
 
 ```
-  def format_url(url)
-    url_hash = @web_formatter.format_url(url)
-    url_hash.merge!({ verified_url: nil, url_redirected: false, response_code: nil, url_sts: nil, url_date: Time.now, wx_date: nil, timeout: nil })
-    url_hash = evaluate_formatted_url(url_hash)
-  end
-
+[ {:is_reformatted=>false,
+    :url_path=>"https://www.steXXXXXXmitsubishiserviceandpartscenter.com",
+    :formatted_url=>"https://www.steXXXXXXmitsubishiserviceandpartscenter.com",
+    :neg=>["neg_urls: parts, rv, service"],
+    :pos=>["pos_urls: mitsubishi"]},
+
+   {:is_reformatted=>false,
+    :url_path=>"https://www.perXXXXXXchryslerjeepcenterville.com",
+    :formatted_url=>"https://www.perXXXXXXchryslerjeepcenterville.com",
+    :neg=>["neg_urls: rv"],
+    :pos=>["pos_urls: chrysler, jeep"]},
+
+   {:is_reformatted=>false,
+    :url_path=>"http://www.pXXXXXXchryslerjeepcenterville.com",
+    :formatted_url=>"http://www.XXXXXXechryslerjeepcenterville.com",
+    :neg=>["neg_urls: rv"],
+    :pos=>["pos_urls: chrysler, jeep"]},
+
+   {:is_reformatted=>false,
+    :url_path=>"http://www.colXXXXXXchryslerdodgejeepram.com",
+    :formatted_url=>"http://www.colXXXXXXchryslerdodgejeepram.com",
+    :neg=>["neg_urls: rv"],
+    :pos=>["pos_urls: chrysler, dodge, jeep, ram"]}
+  ]
 ```
 
 
-
 ## Author
 
 Adam J Booth  - [4rlm](https://github.com/4rlm)

data/gem_notes_crm_formatter.txt

@@ -1,3 +1,32 @@
+
+
+========= Quick Reference =========
+* delete current gem build record.
+* up count gemspec version
+$ gem build crm_formatter.gemspec
+* Uninstall Locally
+  $ gem list
+  $ gem uninstall crm_formatter
+$ gem install crm_formatter-1.0.6.pre.rc.1.gem
+* Test Locally
+  * Edit Gem Tester App Gemfile 
+
+* git add / commit / push
+$ gem push crm_formatter-1.0.6.pre.rc.1.gem
+
+
+========= TO DO =========
+1. Red Flags: Change to 'OA'
+2. Change Meth from 'remove' to 'compare_oa'
+3. Zip can be integer?
+4. Copy formatted results for README.
+5. Continue and finish README.
+6. Test Changes locally before pushing.
+7. Push.
+=========================
+
+
+
 ==== FIRST CREATE APP TO TEST GEM ====
 -----------------------------------
 rails new gem_tester
@@ -38,9 +67,9 @@ $ gem build crm_formatter.gemspec
   Successfully built RubyGem
   Name: crm_formatter
   Version: 1.0.5.pre.rc.1
-  File: crm_formatter-1.0.5.pre.rc.1.gem
+  File: crm_formatter-1.0.6.pre.rc.1.gem
 -----------------------------------
-$ gem install crm_formatter-1.0.5.pre.rc.1.gem
+$ gem install crm_formatter-1.0.6.pre.rc.1.gem
   Successfully installed crm_formatter-1.0.5.pre.rc.1
   Parsing documentation for crm_formatter-1.0.5.pre.rc.1
   Installing ri documentation for crm_formatter-1.0.5.pre.rc.1
@@ -56,7 +85,7 @@ $ irb
 $ curl -u adamjbooth https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials; chmod 0600 ~/.gem/credentials
   Enter host password for user 'adamjbooth': RG<usual>rg
   -----------------------------------
-$ gem push crm_formatter-1.0.5.pre.rc.1.gem
+$ gem push crm_formatter-1.0.6.pre.rc.1.gem
   Pushing gem to https://rubygems.org...
   Successfully registered gem: crm_formatter (1.0.4.pre.rc.1)
   -----------------------------------

data/lib/crm_formatter/extensions.csv

@@ -0,0 +1 @@
+aaa

data/lib/crm_formatter/version.rb

@@ -1,3 +1,3 @@
 module CRMFormatter
-  VERSION = "1.0.6-rc.1"
+  VERSION = "1.0.7-rc.1"
 end

data/lib/crm_formatter/web.rb

@@ -1,171 +1,309 @@
+require 'csv'
+
 module CRMFormatter
   class Web
 
-      def initialize(args={})
-        @url_flags = args.fetch(:url_flags, [])
-        @link_flags = args.fetch(:link_flags, [])
-        @href_flags = args.fetch(:href_flags, [])
-        @extension_flags = args.fetch(:extension_flags, [])
-        @length_min = args.fetch(:length_min, 2)
-        @length_max = args.fetch(:length_max, 100)
+    def initialize(args={})
+      @empty_oa = args.empty?
+      @pos_urls = args.fetch(:pos_urls, [])
+      @neg_urls = args.fetch(:neg_urls, [])
+      @pos_links = args.fetch(:pos_links, [])
+      @neg_links = args.fetch(:neg_links, [])
+      @pos_hrefs = args.fetch(:pos_hrefs, [])
+      @neg_hrefs = args.fetch(:neg_hrefs, [])
+      @pos_exts = args.fetch(:pos_exts, [])
+      @neg_exts = args.fetch(:neg_exts, [])
+      @min_length = args.fetch(:min_length, 2)
+      @max_length = args.fetch(:max_length, 100)
+    end
+
+    def banned_symbols
+      banned_symbols = ["!", "$", "%", "'", "(", ")", "*", "+", ",", "<", ">", "@", "[", "]", "^", "{", "}", "~"]
+    end
+
+    ##Call: StartCrm.run_webs
+    def format_url(url)
+      prep_result = prep_for_uri(url)
+      url_hash = prep_result[:url_hash]
+      url = prep_result[:url]
+      url = nil if has_errors(url_hash)
+
+      if url.present?
+        uri_result = run_uri(url_hash, url)
+        url_hash = uri_result[:url_hash]
+        url = uri_result[:url]
+        (url = nil if has_errors(url_hash)) if url.present?
+      end
+
+      url_hash[:formatted_url] = url
+      url_hash = check_reformatted_status(url_hash) if url.present?
+      url_hash
+    end
+
+
+    def check_reformatted_status(url_hash)
+      formatted = url_hash[:formatted_url]
+      if formatted.present?
+        url_hash[:is_reformatted] = url_hash[:url_path] != formatted
       end
+      url_hash
+    end
+
+
+    def has_errors(url_hash)
+      errors = url_hash[:neg].map { |neg| neg.include?('error') }
+      errors.any?
+    end
+
+
+    ##Call: StartCrm.run_webs
+    def prep_for_uri(url)
+      url_hash = { is_reformatted: false, url_path: url, formatted_url: nil, neg: [], pos: [] }
+      begin
+        url = url&.split('|')&.first
+        url = url&.split('\\')&.first
+        url&.gsub!(/\P{ASCII}/, '')
+        url = url&.downcase&.strip
+
+        2.times { remove_ww3(url) } if url.present?
+        url = remove_slashes(url) if url.present?
+        url&.strip!
 
-      def format_url(url)
-        url_hsh = {url_path: url, formatted_url: nil, url_edit: false }
         if url.present?
-          begin
-            url = url&.split('|')&.first
-            url = url&.split('\\')&.first
-            url&.gsub!(/\P{ASCII}/, '')
-            url = url&.downcase&.strip
-            return url_hsh if url&.length < @length_min
+          url = nil if url.include?(' ')
+          url = url[0..-2] if url.present? && url[-1] == '/'
+        end
 
-            2.times { remove_ww3(url) } if url.present?
-            url = remove_slashes(url) if url.present?
-            url&.strip!
+        url = nil if url.present? && banned_symbols.any? {|symb| url&.include?(symb) }
 
-            return url_hsh if !url.present? || url&.include?(' ')
-            url = url[0..-2] if url[-1] == '/'
+        if url.present?
+          url_hash = compare_criteria(url_hash, url, 'pos_urls', 'include') if !@empty_oa
+          url_hash = compare_criteria(url_hash, url, 'neg_urls', 'include') if !@empty_oa
+        else
+          url_hash[:neg] << "error: syntax"
+          url_hash[:formatted_url] = url
+        end
 
-            symbs = ['(', ')', '[', ']', '{', '}', '*', '@', '^', '$', '+', '!', '<', '>', '~', ',', "'"]
+      rescue Exception => e
+        url_hash[:neg] << "error: #{e}"
+        url = nil
+        url_hash
+      end
 
-            return url_hsh if symbs.any? {|symb| url&.include?(symb) }
+      prep_result = { url_hash: url_hash, url: url }
+    end
 
-            uri = URI(url)
-            if uri.present?
-              host_parts = uri.host&.split(".")
 
-              if @extension_flags.any?
-                bad_host_sts = host_parts&.map { |part| TRUE if @extension_flags.any? {|ext| part == ext } }&.compact&.first
-                return url_hsh if bad_host_sts
-              end
+    ##Call: StartCrm.run_webs
+    def run_uri(url_hash, url)
+      begin
+        uri = URI(url)
+        host_parts = uri.host&.split(".")
 
-              host = uri.host
-              scheme = uri.scheme
-              url = "#{scheme}://#{host}" if host.present? && scheme.present?
-              url = "http://#{url}" if url[0..3] != "http"
-              url = url.gsub("//", "//www.") if !url.include?("www.")
+        url_hash = compare_criteria(url_hash, host_parts, 'pos_exts', 'equal') if !@empty_oa
+        url_hash = compare_criteria(url_hash, host_parts, 'neg_exts', 'equal') if !@empty_oa
 
-              return url_hsh if @url_flags.any? { |bad_text| url&.include?(bad_text) }
+        host = uri.host
+        scheme = uri.scheme
+        url = "#{scheme}://#{host}" if host.present? && scheme.present?
+        url = "http://#{url}" if url[0..3] != "http"
+        url = url.gsub("//", "//www.") if !url.include?("www.")
+        samp_url = convert_to_scheme_host(url)
 
-              url_hsh[:formatted_url] = convert_to_scheme_host(url) if url.present?
-              url_hsh[:url_edit] = url_hsh[:formatted_url] != url_hsh[:url_path]
-            end
-          rescue
-            return url_hsh
+        url = convert_to_scheme_host(url) if url.present?
+        url_extens_result = check_url_extens(url_hash, url)
+        url_hash = url_extens_result[:url_hash]
+        url = url_extens_result[:url]
+
+      rescue Exception => e
+        url_hash[:neg] << "error: #{e}"
+        url = nil
+        url_hash
+      end
+
+      uri_result = { url_hash: url_hash, url: url }
+    end
+
+
+    #Source: http://www.iana.org/domains/root/db
+    #Text: http://data.iana.org/TLD/tlds-alpha-by-domain.txt
+    def check_url_extens(url_hash, url)
+      if url.present?
+        url_extens = URI(url).host&.split(".")[2..-1]
+        if url_extens.count > 1
+          file_path = "./lib/crm_formatter/extensions.csv"
+          extens_list = CSV.read(file_path).flatten
+          valid_url_extens = extens_list & url_extens
+
+          if valid_url_extens.count != 1
+            extens_str = valid_url_extens.map { |ext| ".#{ext}" }.join(', ')
+            url_hash[:neg] << "error: exts.count > 1 [#{extens_str}]"
+            url = nil
           end
         end
-        url_hsh
       end
 
+      url_hash[:formatted_url] = url
+      url_extens_result = {url_hash: url_hash, url: url}
+    end
 
-      ###### Supporting Methods Below #######
 
-      #CALL: Formatter.new.remove_ww3(url)
-      def remove_ww3(url)
-        if url.present?
-          url.split('.').map { |part| url.gsub!(part,'www') if part.scan(/ww[0-9]/).any? }
-          url&.gsub!("www.www", "www")
+    ## This process, compare_criteria only runs if client OA args were passed at initialization.
+    ## Results listed in url_hash[:neg]/[:pos], and don't impact or hinder final formatted url.
+    ## Simply adds more details about user's preferences and criteria for the url are.
+
+    def compare_criteria(hash, target, list_name, include_or_equal)
+      unless @empty_oa
+        if list_name.present?
+          criteria_list = instance_variable_get("@#{list_name}")
+
+          if criteria_list.present?
+            if target.is_a?(::String)
+              tars = target.split(', ')
+            else
+              tars = target
+            end
+
+            pn_matches = tars.map do |tar|
+              if criteria_list.present?
+                if include_or_equal == 'include'
+                  criteria_list.select { |el| el if tar.include?(el) }.join(', ')
+                elsif include_or_equal == 'equal'
+                  criteria_list.select { |el| el if tar == el }.join(', ')
+                end
+              end
+            end
+
+            pn_match = pn_matches&.uniq&.sort&.join(', ')
+            if pn_match.present?
+              if list_name.include?('neg')
+                hash[:neg] << "#{list_name}: #{pn_match}"
+              else
+                hash[:pos] << "#{list_name}: #{pn_match}"
+              end
+            end
+          end
+
         end
       end
+      
+      hash
+    end
 
+    ###### Supporting Methods Below #######
 
-      # For rare cases w/ urls with mistaken double slash twice.
-      def remove_slashes(url)
-        if url.present? && url.include?('//')
-          parts = url.split('//')
-          return parts[0..1].join if parts.length > 2
-        end
+    def extract_link(url_path)
+      url_hash = format_url(url_path)
+      url = url_hash[:formatted_url]
+      link = url_path
+      link_hsh = {url_path: url_path, url: url, link: nil }
+      if url.present? && link.present? && link.length > @min_length
+        url = strip_down_url(url)
+        link = strip_down_url(link)
+        link&.gsub!(url, '')
+        link = link&.split('.net')&.last
+        link = link&.split('.com')&.last
+        link = link&.split('.org')&.last
+        link = "/#{link.split("/").reject(&:empty?).join("/")}" if link.present?
+        link_hsh[:link] = link if link.present? && link.length > @min_length
+      end
+      link_hsh
+    end
+
+
+    def strip_down_url(url)
+      if url.present?
+        url = url.downcase.strip
+        url = url.gsub('www.', '')
+        url = url.split('://')
+        url = url[-1]
         return url
       end
+    end
 
 
-      def extract_link(url_path)
-        url_hsh = format_url(url_path)
-        url = url_hsh[:formatted_url]
-        link = url_path
-        link_hsh = {url_path: url_path, url: url, link: nil }
-        if url.present? && link.present? && link.length > @length_min
-          url = strip_down_url(url)
-          link = strip_down_url(link)
-          link&.gsub!(url, '')
-          link = link&.split('.net')&.last
-          link = link&.split('.com')&.last
-          link = link&.split('.org')&.last
-          link = "/#{link.split("/").reject(&:empty?).join("/")}" if link.present?
-          link_hsh[:link] = link if link.present? && link.length > @length_min
-        end
-        link_hsh
+    def remove_invalid_links(link)
+      link_hsh = {link: link, valid_link: nil, flags: nil }
+      if link.present?
+        @neg_links += get_symbs
+        flags = @neg_links.select { |red| link&.include?(red) }
+        flags << "below #{@min_length}" if link.length < @min_length
+        flags << "over #{@max_length}" if link.length > @max_length
+        flags = flags.flatten.compact
+        flags.any? ? valid_link = nil : valid_link = link
+        link_hsh[:valid_link] = valid_link
+        link_hsh[:flags] = flags.join(', ')
       end
+      link_hsh
+    end
 
 
-      def strip_down_url(url)
-        if url.present?
-          url = url.downcase.strip
-          url = url.gsub('www.', '')
-          url = url.split('://')
-          url = url[-1]
-          return url
-        end
+    def remove_invalid_hrefs(href)
+      href_hsh = {href: href, valid_href: nil, flags: nil }
+      if href.present?
+        @neg_hrefs += get_symbs
+        href = href.split('|').join(' ')
+        href = href.split('/').join(' ')
+        href&.gsub!("(", ' ')
+        href&.gsub!(")", ' ')
+        href&.gsub!("[", ' ')
+        href&.gsub!("]", ' ')
+        href&.gsub!(",", ' ')
+        href&.gsub!("'", ' ')
+
+        flags = []
+        flags << "over #{@max_length}" if href.length > @max_length
+        invalid_text = Regexp.new(/[0-9]/)
+        flags << invalid_text&.match(href)
+        href = href&.downcase
+        href = href&.strip
+
+        flags << @neg_hrefs.select { |red| href&.include?(red) }
+        flags = flags.flatten.compact.uniq
+        href_hsh[:valid_href] = href unless flags.any?
+        href_hsh[:flags] = flags.join(', ')
       end
+      href_hsh
+    end
 
 
-      def remove_invalid_links(link)
-        link_hsh = {link: link, valid_link: nil, flags: nil }
-        if link.present?
-          invalid_chars = ['(', ')', '[', ']', '{', '}', '*', '@', '^', '$', '%', '+', '!', '<', '>', '~', ',', "'"]
-          @link_flags += invalid_chars
-          flags = @link_flags.select { |red| link&.include?(red) }
-          flags << "below #{@length_min}" if link.length < @length_min
-          flags << "over #{@length_max}" if link.length > @length_max
-          flags = flags.flatten.compact
-          flags.any? ? valid_link = nil : valid_link = link
-          link_hsh[:valid_link] = valid_link
-          link_hsh[:flags] = flags.join(', ')
-        end
-        link_hsh
+    def convert_to_scheme_host(url)
+      if url.present?
+        uri = URI(url)
+        scheme = uri&.scheme
+        host = uri&.host
+        url = "#{scheme}://#{host}" if (scheme.present? && host.present?)
+        return url
       end
+    end
 
 
-      def remove_invalid_hrefs(href)
-        href_hsh = {href: href, valid_href: nil, flags: nil }
-        if href.present?
-          symbs = ['{', '}', '*', '@', '^', '$', '%', '+', '!', '<', '>', '~']
-          @href_flags += symbs
-          href = href.split('|').join(' ')
-          href = href.split('/').join(' ')
-          href&.gsub!("(", ' ')
-          href&.gsub!(")", ' ')
-          href&.gsub!("[", ' ')
-          href&.gsub!("]", ' ')
-          href&.gsub!(",", ' ')
-          href&.gsub!("'", ' ')
-
-          flags = []
-          flags << "over #{@length_max}" if href.length > @length_max
-          invalid_text = Regexp.new(/[0-9]/)
-          flags << invalid_text&.match(href)
-          href = href&.downcase
-          href = href&.strip
-
-          flags << @href_flags.select { |red| href&.include?(red) }
-          flags = flags.flatten.compact.uniq
-          href_hsh[:valid_href] = href unless flags.any?
-          href_hsh[:flags] = flags.join(', ')
-        end
-        href_hsh
+    #CALL: Formatter.new.remove_ww3(url)
+    def remove_ww3(url)
+      if url.present?
+        url.split('.').map { |part| url.gsub!(part,'www') if part.scan(/ww[0-9]/).any? }
+        url&.gsub!("www.www", "www")
       end
+    end
 
 
-      def convert_to_scheme_host(url)
-        if url.present?
-          uri = URI(url)
-          scheme = uri&.scheme
-          host = uri&.host
-          url = "#{scheme}://#{host}" if (scheme.present? && host.present?)
-          return url
-        end
+    # For rare cases w/ urls with mistaken double slash twice.
+    def remove_slashes(url)
+      if url.present? && url.include?('//')
+        parts = url.split('//')
+        return parts[0..1].join if parts.length > 2
       end
+      return url
+    end
+
+    ##Call: StartCrm.run_webs
+    # def get_ext_list
+    #   # Source: http://www.iana.org/domains/root/db
+    #   # .txt list: http://data.iana.org/TLD/tlds-alpha-by-domain.txt
+    #   file_path = "./lib/crm_formatter/extensions.csv"
+    #   extensions = CSV.read(file_path)
+    # end
+
 
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: crm_formatter
 version: !ruby/object:Gem::Version
-  version: 1.0.6.pre.rc.1
+  version: 1.0.7.pre.rc.1
 platform: ruby
 authors:
 - Adam Booth
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-05-14 00:00:00.000000000 Z
+date: 2018-05-15 00:00:00.000000000 Z
 dependencies: []
 description: Reformat and Normalize CRM Contact Data, Addresses, Phones, Emails and
   URLs.  Originally developed for proprietary use in an enterprise software suite.  Recently
@@ -30,6 +30,7 @@ files:
 - gem_notes_crm_formatter.txt
 - lib/crm_formatter.rb
 - lib/crm_formatter/address.rb
+- lib/crm_formatter/extensions.csv
 - lib/crm_formatter/helpers.rb
 - lib/crm_formatter/phone.rb
 - lib/crm_formatter/version.rb