number_normalizer 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: a6d41f5736c2b795bf67f8accbb6ea4f7a8e0548
+  data.tar.gz: ffad646d8150cbb7a33a23286d90f2e3b7e6dfd9
+SHA512:
+  metadata.gz: 20b75ba6963bd2392bcffb625c4519d3f10612d800490d766fa69b9ad5747c562f2a1701821fd82f51416eda1e8d6092a4592fc1ac3414123ef329b17ef84a98
+  data.tar.gz: 813b80035cbb5670c9be080a531c644556e8ed37e90b1c16603fd68c446811b554705668b76559c1aee44e9a2146e82a1ec198aebb2d47d1079f4aca7ae76983

data/.gitignore

@@ -0,0 +1,10 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+.project

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,4 @@
+language: ruby
+rvm:
+  - 2.3.0
+before_install: gem install bundler -v 1.11.2

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,49 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at TODO: Write your email address. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in number_normalizer.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 TODO: Write your name
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,63 @@
+# NumberNormalizer
+
+This gem can help you to extract and normalize numbers from a piece of free text.
+
+__Extraction__
+
+- Numbers expressed as digits in forms of eg.
+  * 1000000
+  * 1 000 000
+  * 1 120 123.12
+- Numbers expressed as plain words, eg.
+  * two hundred and four million one hundred ninety-five thousand --> 204195000
+  * __Note__: currently it supports English only.
+
+__Normalization__
+
+- Currently it will normalize all numbers to digits (Integer or Float).
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'number_normalizer'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install number_normalizer
+
+## Usage
+
+```ruby
+t = "The area is 10 123 456 square meters. The population is 1 300 000 000."
+n = NumberNormalizer.new t
+puts n.digit_numbers  # [10123456, 1300000000]
+```
+
+```ruby
+t = "There are two hundred million five thousand people."
+n = NumberNormalizer.new t
+puts n.digit_numbers  # [200005000]
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/dandanwei/number_normalizer. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "number_normalizer"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/number_normalizer.rb

@@ -0,0 +1,226 @@
+require "number_normalizer/version"
+require "set"
+
+class NumberNormalizer
+
+  attr_reader :text
+  attr_reader :matches
+
+  NUM_WORD = {
+    'zero'       =>  0 ,
+    'one'        =>  1 ,
+    'two'        =>  2 ,
+    'three'      =>  3 ,
+    'four'       =>  4 ,
+    'five'       =>  5 ,
+    'six'        =>  6 ,
+    'seven'      =>  7 ,
+    'eight'      =>  8 ,
+    'nine'       =>  9 ,
+    'ten'        =>  10,
+    'eleven'     =>  11,
+    'twelve'     =>  12,
+    'thirteen'   =>  13,
+    'fourteen'   =>  14,
+    'fifteen'    =>  15,
+    'sixteen'    =>  16,
+    'seventeen'  =>  17,
+    'eighteen'   =>  18,
+    'nineteen'   =>  19,
+    'twenty'     =>  20,
+    'thirty'     =>  30,
+    'forty'      =>  40,
+    'fifty'      =>  50,
+    'sixty'      =>  60,
+    'seventy'    =>  70,
+    'eighty'     =>  80,
+    'ninety'     =>  90,
+    'a'          =>  1,
+    'an'         =>  1,
+  }
+
+  MULT_WORD = {
+    'half'       => 0.5,
+    'dozen'      => 12,
+    'hundred'    => 100,
+    'thousand'   => 1000,
+    'million'    => 1000000,
+    'billion'    => 1000000000,
+    'trillion'   => 1000000000000,
+  }
+
+  ADJ_WORD = {
+    '-'          => 0,
+    'and'        => 0,
+  }
+
+  def initialize text
+    @text =  text
+    @token = ' '
+    @text_array = []
+    # parse all numbers and store it in an array
+    # [ {original_format => "1 thousand",
+    #    digit_format => "1000",
+    #    word_format => "one thousand",
+    #    occurance => 1}, ... ]
+    @p_digits_only = '\d+'
+    @p_floating_digits = '\d+\.\d+'
+    @p_floating_digits_2 = '\.\d+'
+    @p_space_sperated_digits = '\d{,3}(\s\d{3}){1,}(\.\d{1,3}){,1}(\s\d{3})*(\s\d{,3})?'
+    @p_comma_sperated_digits = '\d{,3}(,\d{3}){1,}(\.\d{1,3}){,1}(,\d{3})*(,\d{,3})?'
+    @p_begin = ''
+    @p_end = '\W?(\s|$)'
+    @matches = {}
+
+    find_all_digits
+
+    find_all_words
+  end
+
+  def digit_numbers
+
+    convert_digit_string_to_numbers
+    digits = Set.new
+
+    #puts @matches
+
+    @matches.each_value do |v|
+      if v.has_key?:digit_form
+        digits.add(v[:digit_form])
+      end
+    end
+
+    return digits.to_a
+  end
+
+protected
+
+  def tokenlize
+    @text_array = @text.split(@token)
+  end
+
+  def find_all_digits
+    digits_pattern_str = '(' + @p_space_sperated_digits + '|' \
+                         + @p_comma_sperated_digits + '|' \
+                         + @p_floating_digits + '|' \
+                         + @p_floating_digits_2 + '|' \
+                         + @p_digits_only + ')' \
+                         + @p_end
+
+    digits_pattern = Regexp.new(digits_pattern_str)
+
+    @text.enum_for(:scan, digits_pattern).each do |m|
+      mstr = m[0]
+      position = Regexp.last_match.begin(0)
+      if @matches.has_key?mstr
+        @matches[mstr][:pos].push(position)
+      else
+        @matches[mstr] = {:pos => [position],
+                          :type => :digits}
+      end
+    end
+  end
+
+  def find_all_words
+    text_arr = preprocess(@text)
+
+    num_words = ''
+    temp_words = ''
+    temp_num = 0
+    temp_mode = 0
+    hundred_num = 0
+    final_num = 0
+    flag = false
+    is_prev_mult = false
+    is_prev_num = false
+
+    text_arr.each do |str|
+      num = NUM_WORD[str]
+      mult = MULT_WORD[str]
+      adj = ADJ_WORD[str]
+      #puts "#{str}==#{num}==#{mult}==#{adj}=="
+
+      if num != nil
+        if temp_mode > 100
+          temp_num += num
+          temp_words = temp_words + ' ' + str
+        else
+          hundred_num += num
+          num_words = num_words + ' ' + str
+          temp_mode = 0
+        end
+        flag = true
+        is_prev_mult = false
+        is_prev_num = true
+      elsif flag  && adj != nil && ((str == '-') || (is_prev_mult && (str == 'and')))
+        if temp_mode >= 100 && temp_mode % 10 == 0 && str == 'and'
+          temp_mode += 1
+        end
+        num_words = num_words + ' ' + str if str == 'and'
+        num_words += str if str == '-'
+        is_prev_mult = false
+        is_prev_num = false
+      elsif flag && mult != nil
+        if (temp_mode > 1000 && temp_mode % 10 == 1) || (temp_mode > 100 && temp_mode < 1000 && mult < 1000)
+          # 2 numbers
+          # -- save previous number first
+          final_num += hundred_num
+          save_num_words(num_words.strip, final_num)
+          # -- start counting current number
+          num_words = temp_words
+          hundred_num = temp_num
+          temp_words = ''
+          temp_num = 0
+          final_num = 0
+        else
+          # 1 number
+          hundred_num += temp_num
+          temp_num = 0
+        end
+        hundred_num *= mult
+        if hundred_num >= 1000
+          final_num += hundred_num
+          hundred_num = 0
+        end
+        num_words = num_words + ' ' + str
+        is_prev_mult = true
+        is_prev_num = false
+        temp_mode = mult
+      elsif flag
+        final_num += hundred_num + temp_num
+        save_num_words(num_words.strip, final_num)
+        num_words = ''
+        final_num = 0
+        hundred_num = 0
+        temp_num = 0
+        temp_words = ''
+        temp_mode = 0
+        flag = false
+        is_prev_mult = false
+        is_prev_num = false
+      end
+    end
+
+    save_num_words(num_words.strip, final_num + hundred_num) if flag
+  end
+
+  def save_num_words(num_words, number)
+    @matches[num_words] = {:pos => [1],
+                          :type => :words,
+                          :digit_form => number}
+  end
+
+  def convert_digit_string_to_numbers
+    @matches.each_pair do |key, value|
+      if value[:type] == :digits
+        s = key.gsub(/[\s,]/, '')
+        value[:digit_form] = if s =~ /\./ then s.to_f else s.to_i end
+      end
+    end
+  end
+
+  def preprocess(str)
+    str.downcase.gsub(/([^a-zA-Z0-9\s])/, ' \1 ').split()
+  end
+
+end

data/lib/number_normalizer/version.rb

@@ -0,0 +1,3 @@
+class NumberNormalizer
+  VERSION = "0.1.1"
+end

data/number_normalizer.gemspec

@@ -0,0 +1,27 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'number_normalizer/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "number_normalizer"
+  spec.version       = NumberNormalizer::VERSION
+  spec.authors       = ["Dandan Wei"]
+  spec.email         = ["dandan.wei@ymail.com"]
+
+  spec.summary       = %q{Normalize and extract numbers writen as words or digits or mixed in a text}
+  spec.description   = "In free texts, numbers maybe writen as digits (eg. 123.4) or words (eg. one hundred fifty) " \
+                       "or a mixed way (eg. 1 million). This gem will help you to normalize or extract the numbers " \
+                       "presented in the free text."
+  spec.homepage      = "https://github.com/dandanwei/number_normalizer"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.10"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.0"
+end

metadata

@@ -0,0 +1,101 @@
+--- !ruby/object:Gem::Specification
+name: number_normalizer
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- Dandan Wei
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2016-08-16 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+description: In free texts, numbers maybe writen as digits (eg. 123.4) or words (eg.
+  one hundred fifty) or a mixed way (eg. 1 million). This gem will help you to normalize
+  or extract the numbers presented in the free text.
+email:
+- dandan.wei@ymail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- CODE_OF_CONDUCT.md
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lib/number_normalizer.rb
+- lib/number_normalizer/version.rb
+- number_normalizer.gemspec
+homepage: https://github.com/dandanwei/number_normalizer
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.2.2
+signing_key: 
+specification_version: 4
+summary: Normalize and extract numbers writen as words or digits or mixed in a text
+test_files: []