ramparts 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 9623918d77fd13b5006a4436fd147eb2cbe3c790
+  data.tar.gz: cc81eb43b0c2378822b9bce67e35f7f6951c609d
+SHA512:
+  metadata.gz: 3532ef87767a9649195d6508284d1cc5603cbc3aaa7e8e7a3809c1db748bf36e0835a9591713418cd5c548f1348c128fca9f6965fb5fa2a4690155cc23a565ad
+  data.tar.gz: 13cf057ea279241cabce865dd559db6178820fc9b4b44039f38bc30ae6101d1d2d3721094c2eddd45a28ded47bb7572219baabbbbaab3464cd4333f3e54d07d3

data/.github/stale.yml

@@ -0,0 +1,24 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 60
+# Number of days of inactivity before a stale Issue or Pull Request is closed
+daysUntilClose: 7
+# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable
+exemptLabels:
+  - pinned
+  - security
+  - "[Status] Maybe Later"
+# Label to use when marking as stale
+staleLabel: wontfix
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Thank you
+  for your contributions.
+# Comment to post when removing the stale label. Set to `false` to disable
+unmarkComment: false
+# Comment to post when closing a stale Issue or Pull Request. Set to `false` to disable
+closeComment: false
+# Limit to only `issues` or `pulls`
+# only: issues

data/.gitignore

@@ -0,0 +1 @@
+coverage

data/.rspec

@@ -0,0 +1 @@
+--require spec_helper

data/.rubocop.yml

@@ -0,0 +1,54 @@
+AllCops:
+  TargetRubyVersion: 2.3
+  Exclude:
+    - 'ramparts.gemspec'
+    - 'Gemfile'
+
+ClassLength:
+  Enabled: false
+
+CyclomaticComplexity:
+  Enabled: false
+
+Documentation:
+  Enabled: false
+
+Metrics/AbcSize:
+  Enabled: false
+
+Lint/AssignmentInCondition:
+  Enabled: false
+
+Style/ParenthesesAroundCondition:
+  Enabled: false
+
+Style/DoubleNegation:
+  Enabled: false
+
+Style/ConditionalAssignment:
+  Enabled: false
+
+Style/StringLiterals:
+  Exclude:
+    - 'spec/data/**/*'
+
+Layout/SpaceAfterComma:
+  Exclude:
+    - 'spec/data/**/*'
+
+Style/WordArray:
+  Exclude:
+    - 'spec/data/**/*'
+
+LineLength:
+  Max: 120
+  Exclude:
+    - 'spec/data/**/*'
+
+BlockLength:
+  Exclude:
+    - 'spec/**/*'
+
+MethodLength:
+  CountComments: false
+  Max: 100

data/.travis.yml

@@ -0,0 +1,8 @@
+language: ruby
+rvm: 2.3.4
+cache:
+  bundler: true
+
+script:
+  - git diff --diff-filter=ACMRTUXB --name-only origin/master... | grep -E '\.rb$' | xargs --no-run-if-empty bundle exec rubocop
+  - bundle exec rspec ./spec/

data/CHANGELOG.md

@@ -0,0 +1,18 @@
+# Change Log
+
+## 0.2.1 (14/12/2017)
+- Creating a `spec_helper.rb` and implement SimpleCov tool
+- Refactored test architecture to use hashes instead of arrays
+- Reached 100% code coverage
+
+## 0.2.0 (12/12/2017)
+- Implementation of multi functions that allows parsing of both email and phone in same method
+
+## 0.1.0 (09/12/2017)
+- Implementation of all basic functions for emails, urls, and phone numbers
+- Initial draft release
+
+## 0.0.4 (08/12/2017)
+- Use Ruby that's already installed on TravisCI
+- Implement [Stale](https://github.com/probot/stale) for issue
+- Add error messages, especially for malformed input

data/CONTRIBUTING.md

@@ -0,0 +1,46 @@
+# Contributing
+
+Everyone is welcome to contribute to Ramparts. Contributing doesn’t just mean submitting pull requests—there are many different ways for you to get involved, including answering questions, reporting or triaging [issues](https://github.com/CareGuide/ramparts/issues), et al.
+
+No matter how you want to get involved, we ask that you be kind, and treat others on the project with respect.
+
+We love pull requests. We'd like to at least comment on, if not
+accept, pull requests within a few days. We may suggest some changes or improvements or alternatives.
+
+Some things that will increase the chance that your pull request is accepted:
+
+* Make sure the tests pass (this includes linting)
+* Update the documentation: code comments, example code, guides. Basically,
+  update everything affected by your contribution.
+* Include any information that would be relevant to reproducing bugs, use cases for new features, etc.
+* If the change does break compatibility, how can it be updated to become backwards compatible, while directing users to the new way of doing things?
+* A suitable and well thought throught commit message
+
+# Branch Naming
+
+Please follow the following naming convention for branch names. Use `-` to separate words.
+
+- Use `feature/...` for feature related changes
+- Use `fix/...` for changes that fix a bug
+- Use `refactor/...` for general refactors of the code (eg. cleaning the code, adding comments)
+- Finally use `update/...` for general updates that aren't refactors or bug fixes but aren't major enough to fall under features
+
+# Testing
+
+Run linting tests on your branch simply by typing `rubocop` when within the top level directory
+
+Run general unit tests simply by typing `rspec` when within the top level directory
+
+When writing tests please have the [first three digits](https://en.wikipedia.org/wiki/555_(telephone_number)) (not the area code) of phone numbers as `555` (or the phonetic/l33t equivalent) to avoid collisions with actual phone numbers
+
+Please also use `example` for the domain of test email addresses for a similar reason
+
+# Stale issue and pull request policy
+
+Issues and pull requests have a shelf life and sometimes they are no longer relevant. All issues and pull requests that have not had any activity for 60 days will be marked as `stale`. Simply leave a comment with information about why it may still be relevant to keep it open. If no activity occurs in the next 7 days, the issue will be automatically closed. Stale PR's will be closed manually.
+
+The goal of this process is to keep the list of open issues and pull requests focused on work that is actionable and important for the maintainers and the community.
+
+# Pull Request Reviews & releasing
+
+Once pull request has been given approval, leadership will realease and deploy the gem (See [Roadmap](https://github.com/CareGuide/ramparts/blob/master/ROADMAP.md)) for automatic deploys)

data/Gemfile

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+source 'http://rubygems.org'
+
+group :test do
+  gem 'rspec'
+  gem 'rubocop'
+  gem 'simplecov', :require => false
+end

data/Gemfile.lock

@@ -0,0 +1,52 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    ast (2.3.0)
+    diff-lcs (1.3)
+    docile (1.1.5)
+    json (2.1.0)
+    parallel (1.12.0)
+    parser (2.4.0.2)
+      ast (~> 2.3)
+    powerpack (0.1.1)
+    rainbow (2.2.2)
+      rake
+    rake (12.3.0)
+    rspec (3.7.0)
+      rspec-core (~> 3.7.0)
+      rspec-expectations (~> 3.7.0)
+      rspec-mocks (~> 3.7.0)
+    rspec-core (3.7.0)
+      rspec-support (~> 3.7.0)
+    rspec-expectations (3.7.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.7.0)
+    rspec-mocks (3.7.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.7.0)
+    rspec-support (3.7.0)
+    rubocop (0.51.0)
+      parallel (~> 1.10)
+      parser (>= 2.3.3.1, < 3.0)
+      powerpack (~> 0.1)
+      rainbow (>= 2.2.2, < 3.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (~> 1.0, >= 1.0.1)
+    ruby-progressbar (1.9.0)
+    simplecov (0.15.1)
+      docile (~> 1.1.0)
+      json (>= 1.8, < 3)
+      simplecov-html (~> 0.10.0)
+    simplecov-html (0.10.2)
+    unicode-display_width (1.3.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  rspec
+  rubocop
+  simplecov
+
+BUNDLED WITH
+   1.16.0

data/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 CareGuide
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,226 @@
+# Ramparts - Spam Detection
+Parses blocks of text to find phone numbers (including phonetic numbers), emails, and spammer urls
+
+## Example
+
+Find obfuscated phone numbers
+
+```
+>> message = "Contact me directly ( FOUR ONE FIVE E I G H T 9 FOUR TWO EIGHT SIX FIVE  ). Hope you cracked that number code."
+>> Ramparts.find_phone_numbers(message)
+[{start_offset: 22, end_offset: 71, type: :phone, value: 'FOUR ONE FIVE E I G H T 9 FOUR TOO EIGHT SIX FIVE'}]
+```
+
+Find obfuscated emails.
+
+```
+>> message = "Looking for honest worker .. contact ashley73299 AT yahoo dot com for more info"
+>> Ramparts.find_emails(message)
+[{start_offset: 37, end_offset: 65, type: :email, value: 'ashley73299 AT yahoo dot com'}]
+```
+
+Find both obfuscated emails and phone numbers.
+
+```
+>> message = "Looking for honest worker .. contact ashley73299 AT yahoo dot com or FOUR FIVE ONE 456 8900 for more info"
+>> Ramparts.find_phone_numbers_and_emails(message)
+[{start_offset: 37, end_offset: 65, type: :email, value: 'ashley73299 AT yahoo dot com'}, {start_offset: 70, end_offset: 92, type: :phone, value: 'FOUR FIVE ONE 456 8900'}]
+```
+
+Count the occurrences of well known spam URLs and keywords
+
+```
+>> message = ""cialis vs viagra spam guestbook.php?action=http://cialiswalmart.shop""
+>> Ramparts.count_urls(message)
+3
+```
+
+## API
+
+#### count_phone_numbers(text, options = {})
+- Returns the count of the number of phone numbers in the text. Currently uses a map reduce paradigm,
+  which incurs information loss but is cleaner to implement, achieves better results, and is 
+  **~2x faster** than `find_phone_numbers`
+- **Input:**
+    - text **[String]** 
+    - options **[Hash]**
+        - parse_leet **[Boolean][Default → True]**
+            - Parses phone numbers that contain l33t syntax. With this set to true eg. `FivE 4 3 F0r On3 67 NiN3` would be caught.
+        - remove_spaces **[Boolean][Default → True]**
+            - Parses phone numbers that contain spaces between the numbers. With this set to true eg. `F i v E 4 3 F 0 r O n 3 67 N i N 3` would be caught.
+- **Output:** 
+    - number of occurrences of phone numbers **[Integer]**
+- **Example**
+    - **Input:** 
+        - text → `"If you're interested in this position, do contact me directly on my phone number ( FOUR ONE FIVE E I G H T 9 FOUR TWO EIGHT SIX FIVE  ). Hope you cracked that number code."`
+    - **Output:** `1`
+    
+#### find_phone_numbers(text, options = {})
+- **Description:** Finds all occurrences of emails within a block of text. Even when l33t speak, phonetics and 
+space variations are used.
+- **Input:**
+    - text **[String]** 
+    - options **[Hash]**
+        - To Be Implemented
+- **Output:** 
+    - **[Array]**
+        - match **[Hash]**
+            - offset: **[Integer]**
+            - value: **[String]**
+- **Example**
+    - **Input:** 
+        - text → `"If you're interested in this position, do contact me directly on my phone number ( FOUR ONE FIVE E I G H T 9 FOUR TWO EIGHT SIX FIVE  ). Hope you cracked that number code."`
+    - **Output:** `[{start_offset: 84, end_offset: 133, type: :phone, value: 'FOUR ONE FIVE E I G H T 9 FOUR TOO EIGHT SIX FIVE'}]`
+
+#### replace_phone_numbers(text, options = {}, &block)
+- **Description:** Replaces all the occurrences of phone numbers within the text with what is returned in the block. Returns the redacted text.
+of text.
+- **Input:**
+    - text **[String]** 
+    - insertable **[String]**
+    - options **[Hash]**
+        - To Be Implemented
+- **Output:** 
+    - updated text **[String]**
+- **Example**
+    - **Usage:** `altered_text = replace_phone_numbers(...) do CENSORED end`
+    - **Input:** 
+        - text → `"If you're interested in this position, do contact me directly on my phone number ( FOUR ONE FIVE E I G H T 9 FOUR TWO EIGHT SIX FIVE  ). Hope you cracked that number code."`
+    - **Output:** `"If you're interested in this position, do contact me directly on my phone number ( CENSORED  ). Hope you cracked that number code."`
+
+#### count_emails(text, options = {})
+- **Description:** Returns the count of the number of emails in the text. Currently uses a map reduce paradigm,
+ which incurs information loss but is cleaner to implement, achieves better results, and is **~2x faster**
+ than `find_emails`
+ - **Input:**
+     - text **[String]** 
+     - options **[Hash]**
+         - aggressive **[Boolean] [Default → `False`]**
+            - doesn't require a `.` or `dot` + a TLD at the end, but instead compares the last word against a well known list of email domains (eg. `contact ashley @ yandex for more info` would be caught)
+ - **Output:** 
+     - number of occurences of emails **[Integer]**
+ - **Example**
+     - **Input:** 
+         - text → `"Hi, Are you seriously interested ..Looking for honest worker .. My e-mail is ashley73299 AT yahoo dot com, I repeat ashley73299 @ yahoo . com ?.. Ashley"`
+     - **Output:** `2`
+ 
+
+#### find_emails(text, options = {})
+- **Description:** Finds all occurrences of emails within a block of text. Even when l33t speak, phonetics are used.
+- **Input:**
+    - text **[String]** 
+    - options **[Hash]**
+        - aggressive **[Boolean] [Default → `False`]**
+            - doesn't require a `.` or `dot` + a TLD at the end, but instead compares the last word against a well known list of email domains (eg. `contact ashley @ yandex for more info` would be caught)
+        - check_for_at **[Boolean] [Default → `False`]**
+            - checks for the word 'at' as '@', currently can result in algorithm being overly greedy as 'at' is such a common word
+- **Output:**
+    - **[Array]**
+        - match **[Hash]**
+            - offset: **[Integer]**
+            - value: **[String]**
+- **Example**
+    - **Input:** 
+        - text → `"Hi, Are you seriously interested ..Looking for honest worker .. My e-mail is ashley73299 AT yahoo dot com, I repeat ashley73299 @ yahoo . com ?.. Ashley"`
+    - **Output:** `[{start_offset: 78, end_offset: 106, type: :email, value: 'ashley73299 AT yahoo dot com'}, {start_offset: 118, end_offset: 143, type: :email, value: 'ashley73299 @ yahoo . com'}]`
+
+#### replace_emails(text, options = {}, &block)
+- **Description:** Replaces all the occurrences of emails within the text with what is returned in the block. Returns the redacted text
+of text.
+- **Input:**
+    - text **[String]**
+    - options **[Hash]**
+         - aggressive **[Boolean] [Default → `False`]**
+            - doesn't require a `.` or `dot` + a TLD at the end, but instead compares the last word against a well known list of email domains (eg. `contact ashley @ yandex for more info` would be caught)
+         - check_for_at **[Boolean] [Default → `False`]**
+             - checks for the word 'at' as '@', currently can result in algorithm being overly greedy as 'at' is such a common word
+- **Output:** 
+    - updated text **[String]**   
+- **Example**
+    - **Usage:** `altered_text = replace_emails(...) do CENSORED end`
+    - **Input:** 
+        - text → `"My name is Cynthia, a friend of mine needs a nanny to watch her baby in your area, her contact is ( jbush042@gmail.com ) She will be waiting to hear from you kindly send her an email now!"`
+    - **Output:** `My name is Cynthia, a friend of mine needs a nanny to watch her baby in your area, her contact is ( CENSORED ) She will be waiting to hear from you kindly send her an email now!`
+
+#### count_phone_numbers_and_emails(text, options = {})
+- **Description:** Returns the count of the number of emails in the text. Currently uses a map reduce paradigm,
+ which incurs information loss but is cleaner to implement, achieves better results, and is **~2x faster**
+ than `find_emails`
+ - **Input:**
+     - text **[String]** 
+     - options **[Hash]**
+         - parse_leet **[Boolean][Default → True]**
+            - Parses phone numbers that contain l33t syntax. With this set to true eg. `FivE 4 3 F0r On3 67 NiN3` would be caught.
+         - remove_spaces **[Boolean][Default → True]**
+            - Parses phone numbers that contain spaces between the numbers. With this set to true eg. `F i v E 4 3 F 0 r O n 3 67 N i N 3` would be caught.
+         - aggressive **[Boolean] [Default → `False`]**
+            - doesn't require a `.` or `dot` + a TLD at the end, but instead compares the last word against a well known list of email domains (eg. `contact ashley @ yandex for more info` would be caught)
+         - check_for_at **[Boolean] [Default → `False`]**
+            - checks for the word 'at' as '@', currently can result in algorithm being overly greedy as 'at' is such a common word
+ - **Output:** 
+     - number of occurences of emails **[Integer]**
+ - **Example**
+     - **Input:** 
+         - text → `"Hi, Are you seriously interested ..Looking for honest worker .. My e-mail is ashley73299 AT yahoo dot com, phone 416 090 78 NINE 5 ?.. Ashley"`
+     - **Output:** `2`
+ 
+
+#### find_phone_numbers_and_emails(text, options = {})
+- **Description:** Finds all occurrences of phone numbers and emails within a block of text.
+- **Input:**
+    - text **[String]** 
+    - options **[Hash]**
+        - parse_leet **[Boolean][Default → True]**
+            - Parses phone numbers that contain l33t syntax. With this set to true eg. `FivE 4 3 F0r On3 67 NiN3` would be caught.
+        - remove_spaces **[Boolean][Default → True]**
+            - Parses phone numbers that contain spaces between the numbers. With this set to true eg. `F i v E 4 3 F 0 r O n 3 67 N i N 3` would be caught.
+        - aggressive **[Boolean] [Default → `False`]**
+            - doesn't require a `.` or `dot` + a TLD at the end, but instead compares the last word against a well known list of email domains (eg. `contact ashley @ yandex for more info` would be caught)
+        - check_for_at **[Boolean] [Default → `False`]**
+            - checks for the word 'at' as '@', currently can result in algorithm being overly greedy as 'at' is such a common word
+ - **Output:**
+     - **[Array]**
+        - match **[Hash]**
+            - offset: **[Integer]**
+            - value: **[String]**
+- **Example**
+    - **Input:** 
+        - text → `"Hi, Are you seriously interested ..Looking for honest worker .. My e-mail is ashley73299 AT yahoo dot com, phone 416 090 78 NINE 5 ?.. Ashley"`
+    - **Output:** `[{start_offset: 78, end_offset: 106, type: :email, value: 'ashley73299 AT yahoo dot com'}, {start_offset: 115, end_offset: 132, type: :phone, value: 'FOUR FIVE ONE 456 8900'}]`
+
+#### replace_phone_numbers_and_emails(text, options = {}, &block)
+- **Description:** Replaces all the occurrences of phone numbers and emails within the text with what is returned from the block. Returns the redacted text
+of text.
+- **Input:**
+    - text **[String]**
+    - options **[Hash]**
+         - parse_leet **[Boolean][Default → True]**
+            - Parses phone numbers that contain l33t syntax. With this set to true eg. `FivE 4 3 F0r On3 67 NiN3` would be caught.
+         - remove_spaces **[Boolean][Default → True]**
+            - Parses phone numbers that contain spaces between the numbers. With this set to true eg. `F i v E 4 3 F 0 r O n 3 67 N i N 3` would be caught.
+         - aggressive **[Boolean] [Default → `False`]**
+            - doesn't require a `.` or `dot` + a TLD at the end, but instead compares the last word against a well known list of email domains (eg. `contact ashley @ yandex for more info` would be caught)
+         - check_for_at **[Boolean] [Default → `False`]**
+             - checks for the word 'at' as '@', currently can result in algorithm being overly greedy as 'at' is such a common word
+- **Output:** 
+    - updated text **[String]**   
+- **Example**
+    - **Usage:** `altered_text = replace_phone_numbers_and_emails(...) do CENSORED end`
+    - **Input:** 
+        - text → `"My name is Cynthia, a friend of mine needs a nanny to watch her baby in your area, her contact is ( jbush042@gmail.com or FOUR FIVE ONE 789 4568 ) She will be waiting to hear from you kindly send her an email now!"`
+    - **Output:** `My name is Cynthia, a friend of mine needs a nanny to watch her baby in your area, her contact is ( CENSORED or CENSORED ) She will be waiting to hear from you kindly send her an email now!`
+
+#### count_urls(text, options = {})
+- **Description:** Simple union regex to find if the text contains bad urls eg. viagra/cialis. Returns a count of the number of occurrences.
+appear in the text.
+- **Input:**
+    - text **[String]** 
+    - options **[Hash]**
+        - To Be Implemented
+ - **Output:** 
+     - number of occurences of matches **[Integer]**
+- **Example**
+    - **Input:** 
+        - text → `"cialis vs cialis spam guestbook.php?action=http://cialiswalmart.shop"`
+    - **Output:** `3`
+