confidential_info_redactor_lite 0.0.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: e843b44fe21d8ebfba53cc804d37e47b68573bdb
+  data.tar.gz: 0f1d5c38bb7c203d0df2137cddde0513fe404e3b
+SHA512:
+  metadata.gz: 745830d5d5c2d6796167d6af93d6035d08d9071ba2297147aae9931f791bf94036818f0c7af74005721a2ba6f7c0be35c4992cb1468dd2dbca4c0ca9f1739829
+  data.tar.gz: 354a0d50bbc246490c39653aad4939f65a3329539ea6da99b8d8d7c608b93f2294e5dcc111bd3a3620219ce31364d22b3178e79b415e90a0707becbb9fe8cacb

data/.gitignore

@@ -0,0 +1,14 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/.rspec

@@ -0,0 +1 @@
+--color

data/.travis.yml

@@ -0,0 +1,5 @@
+language: ruby
+rvm:
+  - "2.1.0"
+  - "2.1.5"
+  - "2.2.0"

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in confidential_info_redactor_lite.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2015 Kevin S. Dias
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,120 @@
+# Confidential Info Redactor Lite
+
+[![Gem Version](https://badge.fury.io/rb/confidential_info_redactor_lite.svg)](http://badge.fury.io/rb/confidential_info_redactor_lite) [![Build Status](https://travis-ci.org/diasks2/confidential_info_redactor_lite.png)](https://travis-ci.org/diasks2/confidential_info_redactor_lite) [![License](https://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat)](https://github.com/diasks2/confidential_info_redactor_lite/blob/master/LICENSE.txt)
+
+
+Confidential Info Redactor Lite is a Ruby gem to semi-automatically redact confidential information from a text. It is the "lite" version of [Confidential Info Redactor](https://github.com/diasks2/confidential_info_redactor) - the difference being that in this gem no languages are included, you need to supply the vocabulary lists for each language you want to use.
+
+This gem is a poor man's named-entity recognition (NER) library built to extract (and later redact) information in a text (such as proper nouns) that may be confidential. 
+
+It differs from typical NER as it makes no attempt to identify whether a token is a person, company, location, etc. It only attempts to extract tokens that might fit into one of those categories.
+
+Your use case may vary, but the gem was written to first extract potential sensitive tokens from a text and then show the user the extracted tokens and let the user decide which ones should be redacted (or add missing tokens to the list).
+
+The way the gem works is rather simple. It uses regular expressions to search for capitalized tokens (1-grams, 2-grams, 3-grams etc.) and then checks whether those tokens match a list of the common vocabulary for that language (e.g. the x most frequent words - the size of x depending on what is available for that language). If the token is not in the list of words for that language it is added to an array of tokens that should be checked by the user as potential "confidential information".
+
+In the sentence "Pepsi and Coca-Cola battled for position in the market." the gem would extract "Pepsi" and "Coca-Cola" as potential tokens to redact.
+
+In addition to searching for proper nouns, the gem also has the functionality to redact numbers, dates, emails and hyperlinks.
+
+## Install  
+
+**Ruby**  
+*Supports Ruby 2.1.0 and above*
+```
+gem install confidential_info_redactor_lite
+```
+
+**Ruby on Rails**  
+Add this line to your application’s Gemfile:
+```ruby
+gem 'confidential_info_redactor_lite'
+```
+
+## Usage
+
+* If no language is specified, the library will default to English.   
+* To specify a language use its two character [ISO 639-1 code](https://www.tm-town.com/languages).  
+
+```ruby
+text = 'Coca-Cola announced a merger with Pepsi that will happen on December 15th, 2020 for $200,000,000,000. Please contact John Smith at j.smith@example.com or visit http://www.super-fake-merger.com.'
+corpus = ['array', 'of', 'common', 'english', 'words']
+tokens = ConfidentialInfoRedactorLite::Extractor.new(text: text, corpus: corpus).extract
+# => ["Coca-Cola", "Pepsi", "John Smith"]
+
+en_dow = %w(monday tuesday wednesday thursday friday saturday sunday) 
+en_dow_abbr = %w(mon tu tue tues wed th thu thur thurs fri sat sun) 
+en_months = %w(january february march april may june july august september october november december) 
+en_month_abbr = %w(jan feb mar apr jun jul aug sep sept oct nov dec) 
+ConfidentialInfoRedactorLite::Redactor.new(text: text, tokens: tokens, dow: en_dow, dow_abbr: en_dow_abbr, months: en_months, months_abbr: en_month_abbr).redact
+# => '<redacted> announced a merger with <redacted> that will happen on <redacted date> for <redacted number>. Please contact <redacted> at <redacted> or visit <redacted>.'
+
+# You can also just use a specific redactor
+ConfidentialInfoRedactorLite::Redactor.new(text: text, dow: en_dow, dow_abbr: en_dow_abbr, months: en_months, months_abbr: en_month_abbr).dates
+# => 'Coca-Cola announced a merger with Pepsi that will happen on <redacted date> for $200,000,000,000. Please contact John Smith at j.smith@example.com or visit http://www.super-fake-merger.com.'
+
+ConfidentialInfoRedactorLite::Redactor.new(text: text, dow: en_dow, dow_abbr: en_dow_abbr, months: en_months, months_abbr: en_month_abbr).numbers
+# => 'Coca-Cola announced a merger with Pepsi that will happen on December 15th, 2020 for <redacted number>. Please contact John Smith at j.smith@example.com or visit http://www.super-fake-merger.com.'
+
+ConfidentialInfoRedactorLite::Redactor.new(text: text, dow: en_dow, dow_abbr: en_dow_abbr, months: en_months, months_abbr: en_month_abbr).emails
+# => 'Coca-Cola announced a merger with Pepsi that will happen on December 15th, 2020 for $200,000,000,000. Please contact John Smith at <redacted> or visit http://www.super-fake-merger.com.'
+
+ConfidentialInfoRedactorLite::Redactor.new(text: text, dow: en_dow, dow_abbr: en_dow_abbr, months: en_months, months_abbr: en_month_abbr).hyperlinks
+# => 'Coca-Cola announced a merger with Pepsi that will happen on December 15th, 2020 for $200,000,000,000. Please contact John Smith at j.smith@example.com or visit <redacted>.'
+
+ConfidentialInfoRedactorLite::Redactor.new(text: text, tokens: tokens, dow: en_dow, dow_abbr: en_dow_abbr, months: en_months, months_abbr: en_month_abbr).proper_nouns
+# => '<redacted> announced a merger with <redacted> that will happen on December 15th, 2020 for $200,000,000,000. Please contact <redacted> at j.smith@example.com or visit http://www.super-fake-merger.com.'
+
+# It is possible to 'turn off' any of the specific redactors
+ConfidentialInfoRedactorLite::Redactor.new(text: text, tokens: tokens, ignore_numbers: true, dow: en_dow, dow_abbr: en_dow_abbr, months: en_months, months_abbr: en_month_abbr).redact
+# => '<redacted> announced a merger with <redacted> that will happen on <redacted date> for $200,000,000,000. Please contact <redacted> at <redacted> or visit <redacted>.'
+
+# It is also possible to change the redaction text
+text = 'Coca-Cola announced a merger with Pepsi that will happen on December 15th, 2020 for $200,000,000,000. Please contact John Smith at j.smith@example.com or visit http://www.super-fake-merger.com.'
+tokens = ['Coca-Cola', 'Pepsi', 'John Smith']
+ConfidentialInfoRedactorLite::Redactor.new(text: text, tokens: tokens, number_text: '**redacted number**', date_text: '^^redacted date^^', token_text: '*****', dow: en_dow, dow_abbr: en_dow_abbr, months: en_months, months_abbr: en_month_abbr).redact
+# => '***** announced a merger with ***** that will happen on ^^redacted date^^ for **redacted number**. Please contact ***** at ***** or visit *****.'
+```
+
+#### Redactor class options
+* `language` *(optional - defaults to 'en' if not specified)*
+* `tokens` *(optional - any tokens to redact from the text)*
+* `number_text` *(optional - change the text for redacted numbers; the standard is `<redacted number>`)*
+* `date_text` *(optional - change the text for redacted dates; the standard is `<redacted date>`)*
+* `token_text` *(optional - change the text for redacted tokens, emails and hyperlinks; the standard is `<redacted>`)*
+* `ignore_emails` *(optional - set to true if you do not want to redact emails)*
+* `ignore_dates` *(optional - set to true if you do not want to redact dates)*
+* `ignore_numbers` *(optional - set to true if you do not want to redact numbers)*
+* `ignore_hyperlinks` *(optional - set to true if you do not want to redact hyperlinks)*
+
+## Contributing
+
+1. Fork it ( https://github.com/diasks2/confidential_info_redactor_lite/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request
+
+## License
+
+The MIT License (MIT)
+
+Copyright (c) 2015 Kevin S. Dias
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,5 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+task :default => :spec

data/confidential_info_redactor_lite.gemspec

@@ -0,0 +1,26 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'confidential_info_redactor_lite/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "confidential_info_redactor_lite"
+  spec.version       = ConfidentialInfoRedactorLite::VERSION
+  spec.authors       = ["Kevin S. Dias"]
+  spec.email         = ["diasks2@gmail.com"]
+  spec.summary       = %q{Semi-automatically redact confidential information from a text (supply your own language packs)}
+  spec.description   = %q{The lite version of https://github.com/diasks2/confidential_info_redactor - include your own language packs.}
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+  spec.required_ruby_version = '>= 2.1.0'
+
+  spec.add_development_dependency "bundler", "~> 1.7"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec"
+  spec.add_runtime_dependency "pragmatic_segmenter"
+end

data/lib/confidential_info_redactor_lite.rb

@@ -0,0 +1,4 @@
+require 'confidential_info_redactor_lite/version'
+require 'confidential_info_redactor_lite/extractor'
+require 'confidential_info_redactor_lite/redactor'
+require 'pragmatic_segmenter'

data/lib/confidential_info_redactor_lite/date.rb

@@ -0,0 +1,149 @@
+module ConfidentialInfoRedactorLite
+  class Date
+    # Rubular: http://rubular.com/r/73CZ2HU0q6
+    DMY_MDY_REGEX = /(\d{1,2}(\/|\.|-)){2}\d{4}/
+
+    # Rubular: http://rubular.com/r/GWbuWXw4t0
+    YMD_YDM_REGEX = /\d{4}(\/|\.|-)(\d{1,2}(\/|\.|-)){2}/
+
+    # Rubular: http://rubular.com/r/SRZ27XNlvR
+    DIGIT_ONLY_YEAR_FIRST_REGEX = /[12]\d{7}\D/
+
+    # Rubular: http://rubular.com/r/mpVSeaKwdY
+    DIGIT_ONLY_YEAR_LAST_REGEX = /\d{4}[12]\d{3}\D/
+
+    attr_reader :string, :dow, :dow_abbr, :months, :months_abbr
+    def initialize(string:, dow:, dow_abbr:, months:, months_abbr:)
+      @string = string
+      @dow = dow
+      @dow_abbr = dow_abbr
+      @months = months
+      @months_abbr = months_abbr
+    end
+
+    def includes_date?
+      long_date || number_only_date
+    end
+
+    def replace
+      new_string = string.dup
+      counter = 0
+      dow_abbr.each do |day|
+        counter +=1 if string.include?('day')
+      end
+      if counter > 0
+        dow_abbr.each do |day|
+          months.each do |month|
+            new_string = new_string.gsub(/#{Regexp.escape(day)}(\.)*(,)*\s#{Regexp.escape(month)}\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+          end
+          months_abbr.each do |month|
+            new_string = new_string.gsub(/#{Regexp.escape(day)}(\.)*(,)*\s#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+          end
+        end
+        dow.each do |day|
+          months.each do |month|
+            new_string = new_string.gsub(/#{Regexp.escape(day)}(,)*\s#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+                                   .gsub(/\d{2}(\.|-|\/)*\s?#{Regexp.escape(month)}(\.|-|\/)*\s?(\d{4}|\d{2})/i, ' <redacted date> ')
+                                   .gsub(/#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+                                   .gsub(/\d{4}\.*\s#{Regexp.escape(month)}\s\d+(rd|th|st)*/i, ' <redacted date> ')
+                                   .gsub(/\d{4}(\.|-|\/)*#{Regexp.escape(month)}(\.|-|\/)*\d+/i, ' <redacted date> ')
+                                   .gsub(/#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*/i, ' <redacted date> ')
+          end
+          months_abbr.each do |month|
+            new_string = new_string.gsub(/#{Regexp.escape(day)}(,)*\s#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+                                   .gsub(/\d{2}(\.|-|\/)*\s?#{Regexp.escape(month)}(\.|-|\/)*\s?(\d{4}|\d{2})/i, ' <redacted date> ')
+                                   .gsub(/#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+                                   .gsub(/\d{4}\.*\s#{Regexp.escape(month)}\s\d+(rd|th|st)*/i, ' <redacted date> ')
+                                   .gsub(/\d{4}(\.|-|\/)*#{Regexp.escape(month)}(\.|-|\/)*\d+/i, ' <redacted date> ')
+                                   .gsub(/#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*/i, ' <redacted date> ')
+          end
+        end
+      else
+        dow.each do |day|
+          months.each do |month|
+            new_string = new_string.gsub(/#{Regexp.escape(day)}(,)*\s#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+                                   .gsub(/\d{2}(\.|-|\/)*\s?#{Regexp.escape(month)}(\.|-|\/)*\s?(\d{4}|\d{2})/i, ' <redacted date> ')
+                                   .gsub(/#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+                                   .gsub(/\d{4}\.*\s#{Regexp.escape(month)}\s\d+(rd|th|st)*/i, ' <redacted date> ')
+                                   .gsub(/\d{4}(\.|-|\/)*#{Regexp.escape(month)}(\.|-|\/)*\d+/i, ' <redacted date> ')
+                                   .gsub(/#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*/i, ' <redacted date> ')
+          end
+          months_abbr.each do |month|
+            new_string = new_string.gsub(/#{Regexp.escape(day)}(,)*\s#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+                                   .gsub(/\d{2}(\.|-|\/)*\s?#{Regexp.escape(month)}(\.|-|\/)*\s?(\d{4}|\d{2})/i, ' <redacted date> ')
+                                   .gsub(/#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+                                   .gsub(/\d{4}\.*\s#{Regexp.escape(month)}\s\d+(rd|th|st)*/i, ' <redacted date> ')
+                                   .gsub(/\d{4}(\.|-|\/)*#{Regexp.escape(month)}(\.|-|\/)*\d+/i, ' <redacted date> ')
+                                   .gsub(/#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*/i, ' <redacted date> ')
+          end
+        end
+        dow_abbr.each do |day|
+          months.each do |month|
+            new_string = new_string.gsub(/#{Regexp.escape(day)}(\.)*(,)*\s#{Regexp.escape(month)}\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+          end
+          months_abbr.each do |month|
+            new_string = new_string.gsub(/#{Regexp.escape(day)}(\.)*(,)*\s#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i, ' <redacted date> ')
+          end
+        end
+      end
+      new_string = new_string.gsub(DMY_MDY_REGEX, ' <redacted date> ')
+                     .gsub(YMD_YDM_REGEX, ' <redacted date> ')
+                     .gsub(DIGIT_ONLY_YEAR_FIRST_REGEX, ' <redacted date> ')
+                     .gsub(DIGIT_ONLY_YEAR_LAST_REGEX, ' <redacted date> ')
+    end
+
+    def occurences
+      replace.scan(/<redacted date>/).size
+    end
+
+    def replace_number_only_date
+      string.gsub(DMY_MDY_REGEX, ' <redacted date> ')
+            .gsub(YMD_YDM_REGEX, ' <redacted date> ')
+            .gsub(DIGIT_ONLY_YEAR_FIRST_REGEX, ' <redacted date> ')
+            .gsub(DIGIT_ONLY_YEAR_LAST_REGEX, ' <redacted date> ')
+    end
+
+    private
+
+    def long_date
+      match_found = false
+      dow.each do |day|
+        months.each do |month|
+          break if match_found
+          match_found = check_for_matches(day, month)
+        end
+        months_abbr.each do |month|
+          break if match_found
+          match_found = check_for_matches(day, month)
+        end
+      end
+      dow_abbr.each do |day|
+        months.each do |month|
+          break if match_found
+          match_found = !(string !~ /#{Regexp.escape(day)}(\.)*(,)*\s#{Regexp.escape(month)}\s\d+(rd|th|st)*(,)*\s\d{4}/i)
+        end
+        months_abbr.each do |month|
+          break if match_found
+          match_found = !(string !~ /#{Regexp.escape(day)}(\.)*(,)*\s#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i)
+        end
+      end
+      match_found
+    end
+
+    def number_only_date
+      !(string !~ DMY_MDY_REGEX) ||
+      !(string !~ YMD_YDM_REGEX) ||
+      !(string !~ DIGIT_ONLY_YEAR_FIRST_REGEX) ||
+      !(string !~ DIGIT_ONLY_YEAR_LAST_REGEX)
+    end
+
+    def check_for_matches(day, month)
+      !(string !~ /#{Regexp.escape(day)}(,)*\s#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i) ||
+      !(string !~ /#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*(,)*\s\d{4}/i) ||
+      !(string !~ /\d{4}\.*\s#{Regexp.escape(month)}\s\d+(rd|th|st)*/i) ||
+      !(string !~ /\d{4}(\.|-|\/)*#{Regexp.escape(month)}(\.|-|\/)*\d+/i) ||
+      !(string !~ /#{Regexp.escape(month)}(\.)*\s\d+(rd|th|st)*/i) ||
+      !(string !~ /\d{2}(\.|-|\/)*#{Regexp.escape(month)}(\.|-|\/)*(\d{4}|\d{2})/i)
+    end
+  end
+end

data/lib/confidential_info_redactor_lite/extractor.rb

@@ -0,0 +1,34 @@
+module ConfidentialInfoRedactorLite
+  # This class extracts proper nouns from a text
+  class Extractor
+    # Rubular: http://rubular.com/r/qE0g4r9zR7
+    EXTRACT_REGEX = /(?<=\s|^|\s\")([A-Z]\S*\s)*[A-Z]\S*(?=(\s|\.|\z))|(?<=\s|^|\s\")[i][A-Z][a-z]+/
+    attr_reader :text, :language, :corpus
+    def initialize(text:, corpus:, **args)
+      @text = text.gsub(/[’‘]/, "'")
+      @corpus = corpus
+      @language = args[:language] || 'en'
+    end
+
+    def extract
+      extracted_terms = []
+      PragmaticSegmenter::Segmenter.new(text: text, language: language).segment.each do |segment|
+        initial_extracted_terms = segment.gsub(EXTRACT_REGEX).map { |match| match unless corpus.include?(match.downcase.gsub(/[\?\.\)\(\!\\\/\"\:\;]/, '').gsub(/\'$/, '')) }.compact
+        initial_extracted_terms.each do |ngram|
+          ngram.split(/[\?\)\(\!\\\/\"\:\;\,]/).each do |t|
+            extracted_terms << t.gsub(/[\?\)\(\!\\\/\"\:\;\,]/, '').gsub(/\'$/, '').gsub(/\.\z/, '').strip unless corpus.include?(t.downcase.gsub(/[\?\.\)\(\!\\\/\"\:\;]/, '').gsub(/\'$/, '').strip)
+          end
+        end
+      end
+
+      if language.eql?('de')
+        extracted_terms.delete_if do |token|
+          corpus.include?(token.split(' ')[0].downcase.strip) &&
+            token.split(' ')[0].downcase.strip != 'deutsche'
+        end.uniq.reject(&:empty?)
+      else
+        extracted_terms.uniq.reject(&:empty?)
+      end
+    end
+  end
+end

data/lib/confidential_info_redactor_lite/hyperlink.rb

@@ -0,0 +1,31 @@
+require 'uri'
+
+module ConfidentialInfoRedactorLite
+  class Hyperlink
+    NON_HYPERLINK_REGEX = /\A\w+:$/
+
+    # Rubular: http://rubular.com/r/fXa4lp0gfS
+    HYPERLINK_REGEX = /(http|https|www)(\.|:)/
+
+    attr_reader :string
+    def initialize(string:)
+      @string = string
+    end
+
+    def hyperlink?
+      !(string !~ URI.regexp) && string !~ NON_HYPERLINK_REGEX && !(string !~ HYPERLINK_REGEX)
+    end
+
+    def replace
+      new_string = string.dup
+      string.split(/\s+/).each do |token|
+        if !(token !~ URI.regexp) && token !~ NON_HYPERLINK_REGEX && !(token !~ HYPERLINK_REGEX) && token.include?('">')
+          new_string = new_string.gsub(/#{Regexp.escape(token.split('">')[0].gsub(/\.\z/, ''))}/, ' <redacted> ')
+        elsif !(token !~ URI.regexp) && token !~ NON_HYPERLINK_REGEX && !(token !~ HYPERLINK_REGEX)
+          new_string = new_string.gsub(/#{Regexp.escape(token.gsub(/\.\z/, ''))}/, ' <redacted> ')
+        end
+      end
+      new_string
+    end
+  end
+end

data/lib/confidential_info_redactor_lite/redactor.rb

@@ -0,0 +1,86 @@
+require 'confidential_info_redactor_lite/date'
+require 'confidential_info_redactor_lite/hyperlink'
+
+module ConfidentialInfoRedactorLite
+  # This class redacts various tokens from a text
+  class Redactor
+    # Rubular: http://rubular.com/r/LRRPtDgJOe
+    NUMBER_REGEX = /(?<=\A|\A\()[^(]?\d+((,|\.)*\d)*(\D?\s|\s|\.?\s|\.$)|(?<=\s|\s\()[^(]?\d+((,|\.)*\d)*(?=(\D?\s|\s|\.?\s|\.$))|(?<=\s)\d+(nd|th|st)|(?<=\s)\d+\/\d+\"*(?=\s)/
+    EMAIL_REGEX = /(?<=\A|\s)[\w+\-.]+@[a-z\d\-]+(\.[a-z]+)*\.[a-z]+(?=\z|\s|\.)/i
+
+    attr_reader :text, :language, :number_text, :date_text, :token_text, :tokens, :ignore_emails, :ignore_dates, :ignore_numbers, :ignore_hyperlinks, :dow, :dow_abbr, :months, :months_abbr
+    def initialize(text:, dow:, dow_abbr:, months:, months_abbr:, **args)
+      @text = text
+      @language = args[:language] || 'en'
+      @tokens = args[:tokens]
+      @number_text = args[:number_text] || '<redacted number>'
+      @date_text = args[:date_text] || '<redacted date>'
+      @token_text = args[:token_text] || '<redacted>'
+      @ignore_emails = args[:ignore_emails]
+      @ignore_dates = args[:ignore_dates]
+      @ignore_numbers = args[:ignore_numbers]
+      @ignore_hyperlinks = args[:ignore_hyperlinks]
+      @dow = dow
+      @dow_abbr = dow_abbr
+      @months = months
+      @months_abbr = months_abbr
+    end
+
+    def dates
+      redact_dates(text)
+    end
+
+    def numbers
+      redact_numbers(text)
+    end
+
+    def emails
+      redact_emails(text)
+    end
+
+    def hyperlinks
+      redact_hyperlinks(text)
+    end
+
+    def proper_nouns
+      redact_tokens(text)
+    end
+
+    def redact
+      if ignore_emails
+        redacted_text = text
+      else
+        redacted_text = redact_emails(text)
+      end
+      redacted_text = redact_hyperlinks(redacted_text) unless ignore_hyperlinks
+      redacted_text = redact_dates(redacted_text) unless ignore_dates
+      redacted_text = redact_numbers(redacted_text) unless ignore_numbers
+      redact_tokens(redacted_text)
+    end
+
+    private
+
+    def redact_hyperlinks(txt)
+      ConfidentialInfoRedactorLite::Hyperlink.new(string: txt).replace.gsub(/<redacted>/, "#{token_text}").gsub(/\s*#{Regexp.escape(token_text)}\s*/, " #{token_text} ").gsub(/#{Regexp.escape(token_text)}\s{1}\.{1}/, "#{token_text}.").gsub(/#{Regexp.escape(token_text)}\s{1}\,{1}/, "#{token_text},")
+    end
+
+    def redact_dates(txt)
+      ConfidentialInfoRedactorLite::Date.new(string: txt, dow: dow, dow_abbr: dow_abbr, months: months, months_abbr: months_abbr).replace.gsub(/<redacted date>/, "#{date_text}").gsub(/\s*#{Regexp.escape(date_text)}\s*/, " #{date_text} ").gsub(/\A\s*#{Regexp.escape(date_text)}\s*/, "#{date_text} ").gsub(/#{Regexp.escape(date_text)}\s{1}\.{1}/, "#{date_text}.")
+    end
+
+    def redact_numbers(txt)
+      txt.gsub(NUMBER_REGEX, " #{number_text} ").gsub(/\s*#{Regexp.escape(number_text)}\s*/, " #{number_text} ").gsub(/\A\s*#{Regexp.escape(number_text)}\s*/, "#{number_text} ").gsub(/#{Regexp.escape(number_text)}\s{1}\.{1}/, "#{number_text}.").gsub(/#{Regexp.escape(number_text)}\s{1}\,{1}/, "#{number_text},").gsub(/#{Regexp.escape(number_text)}\s{1}\){1}/, "#{number_text})").gsub(/\(\s{1}#{Regexp.escape(number_text)}/, "(#{number_text}")
+    end
+
+    def redact_emails(txt)
+      txt.gsub(EMAIL_REGEX, "#{token_text}")
+    end
+
+    def redact_tokens(txt)
+      tokens.sort_by{ |x| x.split.count }.reverse.each do |token|
+        txt.gsub!(/#{Regexp.escape(token)}/, "#{token_text}")
+      end
+      txt
+    end
+  end
+end