utf8_sanitizer 0.0.2.pre.rc.03 → 0.0.2.pre.rc.04

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1128d1d63dfb289df84978713ef4fc7f8efc1bfb7d6784ab7accd85684824bff
-  data.tar.gz: 54111bdd0d5e3e1c6f97892c5404d68cbaf283068f77a3372e3bc6facb7cd9d3
+  metadata.gz: 05155a4029ddd8224888b1482972b0640b4a81ef0990d7bd8f29311a5798d309
+  data.tar.gz: 6d3bd40596087c6e8dfa41934a44ffd2459f6bf738abf77c66aa20ce4591b641
 SHA512:
-  metadata.gz: 0e64c64ab3dafaa19fdc1cac3fae947b8aafe909bde3d747b8a23b8516ad5224769ae690605074a22e85082275e799173e866d58963ef951b62d7183d9503ae7
-  data.tar.gz: '0813f8565e88f697799fd3882ed883595c89d5af7a6042a792b1918fe4e97524ed0ba6da5ac0e1e21b0c91dc7ef89766122000cdeffaaa08505fbfbb300a63c5'
+  metadata.gz: 23d64036dec061d1290d186069adfe018825bbe16a89f733994bce7eb6793fd5f0d402d9cc00c57a590fcb24f6b2f14890cac0611ae54f73bec0cc9b39e060d4
+  data.tar.gz: 972355c619c5c6ef4753df81d7a94f18b4f0e0622555e3fb4c3d227d5a31477ebeea015a10682281ace975bb322daed2752f9cf71123c19b194429893f1b6f78

data/README.md

@@ -1,8 +1,21 @@
 # Utf8Sanitizer
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/utf8_sanitizer`. To experiment with that code, run `bin/console` for an interactive prompt.
+Removes invalid UTF8 characters & extra whitespace (carriage returns, new lines, tabs, spaces, etc.) from csv or strings.
+
+Example:
+```
+"ABC Au\xC1tos, 123 E Main St, Anytown, TX, 78735, (888) 555-1234\n\r\n"
+```
+
+Returns:
+```
+"ABC Autos, 123 E Main St, Anytown, TX, 78735, (888) 555-1234"
+```
+
+Removed:
+Non-UTF8: `\xC1`
+Extra whitespace: `\n\r\n`
 
-TODO: Delete this and the text above, and describe your gem
 
 ## Installation
 
@@ -22,7 +35,84 @@ Or install it yourself as:
 
 ## Usage
 
-TODO: Write usage instructions here
+You have three options for UTF8 Sanitizing your data: CSV Parsing, Data Hash of strings, or run default seed data to test.
+
+#### 1. CSV Parsing
+This is a good option if you are having problems with a CSV containing non-UTF8 characters.  Pass your file_path as a hash like below.  Hash MUST be a SYMBOL and named `:file_path`.  If not, default seeds will be passed as the system detects empty user input and thinks user is trying to run built-in seed data for testing.
+```
+args = {file_path: "./path/to/your_csv.csv"}
+sanitized_data = Utf8Sanitizer.sanitize(args)
+```
+
+#### 2. Hash of Strings
+This is a good option if you are scraping data or cleaning up existing databases.  Pass your data as a hash like below.  Hash MUST be a SYMBOL and named `:data`.  The value of `:data` should be an array of hashes like below and can be any size from one to many tens of thousands.  The hashes inside the data array can be named anything from crm contact data like below, stats, recipes, or any custom hashes as long as they are in an array and resemble the syntax and structure like below.
+```
+data_hash = [ { url: 'abc_autos_example.com',
+           act_name: 'ABC Aut\x92os',
+           street: '123 E Main St\r\n',
+           city: 'Austin',
+           state: 'TX',
+           zip: '78735',
+           phone: '(888) 555-1234\r\n' },
+         { url: 'xyz_trucks_example',
+           act_name: 'XYZ Aut\xC1os',
+           street: '456 W Main St\r\n',
+           city: 'Austin',
+           state: 'TX',
+           zip: '78735',
+           phone: '(800) 555-5678\r\n' },
+      }]
+
+sanitized_data = Utf8Sanitizer.sanitize(data: data_hash)
+```
+
+#### 3. Run Seed Data to Test
+If you want to run built-in seed data to first test, simply run as below without passing args.
+```
+sanitized_data = Utf8Sanitizer.sanitize
+```
+
+### Returned Sanitized Data Format
+The returned data will be in hash format with the following keys: `:stats`, `:file_path`, `:data` like below.  
+
+The `:stats` are a breakdown of the results. `:defective_rows` and `:error_rows` will usually be the same number which refer to the rows which are beyond repair (very rare). Otherwise, the results will be `:valid_rows` if they were perfect or successfully sanitized, including `:encoded_rows` which refers to the number of rows that contained non-utf8 characters, and `:wchar_rows` which is short for 'whitespace character rows'.
+
+`:data` is broken down into the following categories: `:valid_data`, `:encoded_data`, `:defective_data`, and `:error_data`.
+
+`:valid_data` is the most important data and you can access it with `sanitized_data[:data][:valid_data]`.  Each non-UTF8 row will be included in its original syntax like below and can be accessed directly via `sanitized_data[:data][:encoded_data]`.  **You can change the name of `sanitized_data` to anything you like, but it must be followed with `[:data][:valid_data]` and `[:data][:encoded_data]`, etc.**
+
+`:pollute_seeds` is only for running seed data.  It injects each row with non-UTF8 and extra whitespace for testing.  It can be ignored and will only run if your input is nil, which tells the system that you are intentionally trying to run seed data for testing.
+```
+{:stats=>
+  {:total_rows=>2, :header_row=>1, :valid_rows=>2, :error_rows=>0, :defective_rows=>0, :perfect_rows=>0, :encoded_rows=>2, :wchar_rows=>2},
+ :file_path=>nil,
+ :data=>
+  {:valid_data=>
+    [{:row_id=>"1",
+      :utf_status=>"encoded, wchar",
+      :url=>"abc_autos_example.com",
+      :act_name=>"ABC Autos Example",
+      :street=>"123 E Main St",
+      :city=>"Austin",
+      :state=>"TX",
+      :zip=>"78735",
+      :phone=>"(888) 555-1234"},
+     {:row_id=>"2",
+      :utf_status=>"encoded, wchar",
+      :url=>"xyz_trucks_example",
+      :act_name=>"XYZ Trucks Example",
+      :street=>"456 W Main St",
+      :city=>"Austin",
+      :state=>"TX",
+      :zip=>"78735",
+      :phone=>"(800) 555-4321"}],
+   :encoded_data=>
+    [{:row_id=>1, :text=>"1,abc_autos_example.com,ABC Autos Example\x98_\xC0,123 E Main St,Austin,TX,78735,(888) 555-1234\r\n"},
+     {:row_id=>2, :text=>"2,xyz_trucks_example,XYZ \xC1_\xCCTrucks Example,456 W Main St,Austin,TX,78735,(800) 555-4321\r\n"}],
+   :defective_data=>[],
+   :error_data=>[]},
+  :pollute_seeds=>true}
+```
 
 ## Development
 
@@ -32,7 +122,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/utf8_sanitizer. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+Bug reports and pull requests are welcome on GitHub at https://github.com/4rlm/utf8_sanitizer. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
 ## License
 
@@ -40,4 +130,4 @@ The gem is available as open source under the terms of the [MIT License](https:/
 
 ## Code of Conduct
 
-Everyone interacting in the Utf8Sanitizer project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/utf8_sanitizer/blob/master/CODE_OF_CONDUCT.md).
+Everyone interacting in the Utf8Sanitizer project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/4rlm/utf8_sanitizer/blob/master/CODE_OF_CONDUCT.md).

data/lib/utf8_sanitizer/version.rb

@@ -1,4 +1,4 @@
 module Utf8Sanitizer
   # VERSION = "0.0.1-rc.1"
-  VERSION = "0.0.2.pre.rc.03"
+  VERSION = "0.0.2.pre.rc.04"
 end

data/lib/utf8_sanitizer.rb

@@ -21,7 +21,8 @@ module Utf8Sanitizer
     end
 
     ## Sanitizes input hash, then merges results to original input hash, and returns as sanitized_data.
-    input.merge!(Utf8Sanitizer::UTF.new.validate_data(input))
+    sanitized_data = input.merge!(Utf8Sanitizer::UTF.new.validate_data(input))
+    sanitized_data
   end
 
 

data/utf8_sanitizer.gemspec

@@ -15,7 +15,7 @@ Gem::Specification.new do |spec|
   spec.license       = 'MIT'
 
   spec.summary       = "Removes invalid UTF8 characters & extra whitespace from csv or strings."
-  spec.description   = "Removes invalid UTF8 characters & extra whitespace (carriage returns, new lines, tabs, spaces, etc.) from csv, or strings.\n Example: ABC Au\\xC1tos,123 E Main St,Anytown,TX,75142,(888) 555-1234\\n\\r\\n  =>  ABC Autos,123 E Main St,Anytown,TX,75142,(888) 555-1234"
+  spec.description   = "Removes invalid UTF8 characters & extra whitespace (carriage returns, new lines, tabs, spaces, etc.) from csv or strings.\n Example: ABC Au\\xC1tos,123 E Main St,Anytown,TX,75142,(888) 555-1234\\n\\r\\n  =>  ABC Autos,123 E Main St,Anytown,TX,75142,(888) 555-1234"
 
   if spec.respond_to?(:metadata)
     spec.metadata['allowed_push_host'] = "https://rubygems.org"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: utf8_sanitizer
 version: !ruby/object:Gem::Version
-  version: 0.0.2.pre.rc.03
+  version: 0.0.2.pre.rc.04
 platform: ruby
 authors:
 - Adam Booth
@@ -181,7 +181,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: 0.11.3
 description: |-
-  Removes invalid UTF8 characters & extra whitespace (carriage returns, new lines, tabs, spaces, etc.) from csv, or strings.
+  Removes invalid UTF8 characters & extra whitespace (carriage returns, new lines, tabs, spaces, etc.) from csv or strings.
    Example: ABC Au\xC1tos,123 E Main St,Anytown,TX,75142,(888) 555-1234\n\r\n  =>  ABC Autos,123 E Main St,Anytown,TX,75142,(888) 555-1234
 email:
 - 4rlm@protonmail.ch
@@ -205,6 +205,7 @@ files:
 - lib/utf8_sanitizer/csv/seeds_dirty.csv
 - lib/utf8_sanitizer/csv/seeds_mega.csv
 - lib/utf8_sanitizer/csv/seeds_mini.csv
+- lib/utf8_sanitizer/csv/seeds_mini.csv,
 - lib/utf8_sanitizer/csv/seeds_mini_10.csv
 - lib/utf8_sanitizer/csv/seeds_mini_2_bug.csv
 - lib/utf8_sanitizer/seed.rb